Methods and systems for directing movement of a tool in hair transplantation procedures

ABSTRACT

Methods and systems are provided useful in various procedures, including hair harvesting and implantation, and further including computer-implemented and/or robotic hair transplantation. Methodologies are provided which enable selection of follicular unit harvesting or implanting sites. In various cases, such selection may be made using a lowest and closest method, overlap-based methods, position-based methods, pattern-based methods, and/or a combination of these methods. In various combinations of the methods, the output selection from one of the methods may be used as an input for one of the other methods.

RELATED APPLICATIONS

This application is a continuation-in-part of U.S. application Ser. No.13/174,721, filed Jun. 30, 2011 and titled “Methods and Systems forDirecting Movement of a Tool in Hair Transplantation Procedures,” whichin turn claims priority under 35 U.S.C. 119(e) to U.S. ProvisionalApplication No. 61/425,571 filed Dec. 21, 2010, entitled “Methods andSystems for Directing Movement of a Tool in Hair TransplantationProcedures;” the disclosures of which are hereby incorporated byreference, in their entireties.

FIELD OF THE INVENTION

The present application relates generally to hair transplantationprocedures and more particularly to methods and systems used foroperating a tool to harvest or implant follicular units from or into abody surface using imaging and processing techniques.

BACKGROUND OF THE INVENTION

During medical operative procedures on a patient, particularly if theprocedure is of a significant duration of time, it is inevitable thatpatient movement and/or interruptions may occur. These interruptions maybe mechanical, electrical, hardware, software, or medical in nature, orcaused by some other means. For example, it may be desirable or simplyunavoidable that the patient alters his/her position during theprocedure, or that the patient and/or physician temporarily leave theplace in which the operation is being carried out before returning. Thisis relevant to various medical, including cosmetic, procedures, andparticularly relevant, for example, for the case of a patient undergoinga hair transplantation procedure, having follicular units harvested froma donor area (e.g., on the patient's scalp) for transplantation, orhaving follicular units implanted into a recipient area (e.g., a baldarea on the patient's scalp). These procedures typically take several ormore hours to perform. In some instances, the patient may remain in theoperation chair but need to alter their position due to discomfortand/or fatigue, or simply moves due to breathing or other naturalmovements. In other instances the patient may need to interrupt theprocedure to temporarily leave the chair.

SUMMARY OF THE INVENTION

In accordance with one general aspect, the present application disclosessystems and methods for directing movement and operation of a tool inmedical procedures which are at least partially automated. In someembodiments, a method of operating a tool to perform a medical procedureis provided. The method may comprise recording first locations of aplurality of fiducials appearing in one or more images; updating andrecording the updated locations of at least some of the plurality of thefiducials in an updated one or more images; determining an offset of theupdated locations of the at least some of the plurality of the fiducialsrelative to their first locations; selecting a site on which to performthe medical procedure based at least in part on the determined offset.The method may further comprise instructing the tool to perform theprocedure at the selected procedure site (for example, a tattooplacement or tattoo removal procedure, or a cosmetic injectionprocedure, ablation procedure, eye treatment procedure, or any otherprocedure that could benefit from the inventions described herein); andmay also comprise recording a location of the performed medicalprocedure.

In some embodiments the method may further comprise determining aboundary of an area on a body surface where a procedure is performed,for example, an area from which follicular units are to be harvested, orinto which follicular units are to be implanted, and instructing a toolto perform an operation, such as in the example of hair transplantationto harvest from a selected harvesting site or implant into a selectedimplant site, for example, within or outside the determined boundary.The boundary may be determined based on a reference, for example, aplurality of fiducials, which may comprise a set of distinctivefiducials. In some embodiments, with reference to hair transplantation,the selection of follicular unit harvesting or implanting sites may takeinto account limitations of the tool. In those embodiments where theboundary of the area is determined, such boundary may be adjusted toeliminate portions, for example, where a tool used in the procedure haslimited or insufficient access for proper operation, or to take intoaccount one or more parameters of a skin tensioning device, if suchdevice is used, and/or the tool. In the example of the hairtransplantation, the tool may be operated to harvest or implantfollicular units by substantially automatically changing a direction oftravel of the tool based on the locations of the reference points orfiducials. Harvesting and implant sites may be selected based on one ormore criteria, one of which may be to minimize interference from fluidson the body surface. Another such criteria may be selecting locationswhich do not comprise locations of previously harvested or implantedfollicular units.

One embodiment of the method may comprise recording first locations of aplurality of fiducials, and updating and recording updated locations ofat least one or more of the plurality of fiducials, for example, toaccount for movement, whether that be due to interruptions or merelypatient movement. The method may further comprise determining an offsetof the updated locations of at least some of the plurality of fiducialsrelative to the first locations and selecting a procedure site, such asa follicular unit harvesting or implanting site, based at least in parton the determined offset. In some embodiments, the locations from wherefollicular units are harvested or into which follicular units areimplanted are recorded, and a visual representation of the harvested orimplanted follicular unit may be created.

According to another aspect, a system is provided that may include aprocessor comprising a set of instructions for executing operations formoving a tool to perform a procedure, for example, moving a tool to asite where a procedure is to be performed. For example, with referenceto hair transplantation procedure, a set of instructions may compriseinstructions for selecting a procedure site from where follicular unitsare to be harvested (or where follicular units are to be implanted)based on first locations of the plurality of fiducials appearing in oneor more images of the body surface; and updating and recording updatedlocations of at least one or more of the plurality of fiducials. Theinstructions may further provide for a boundary of an area on a bodysurface to be determined, the area from which follicular units can beharvested from or implanted into. The instructions may also provide fordetermining an offset of the updated locations from the first locationsand selecting a procedure site, such as a follicular unit harvesting orimplanting site based at least in part on the determined offset. Theinstructions may further comprise instructing a tool to move to the siteand/or perform the procedure, such as harvest from a selected harvestingsite or implant into a selected implant site, optionally, outside orwithin the determined boundary. The system may comprise an imageacquisition device to provide image data containing one or more imagesof the body surface with fiducials thereon and an interface adapted toreceive an image data containing images of a body surface. The systemmay further comprise a processor configured to create a virtualrepresentation of the site where the procedure has been performed, arepresentation for example of any harvested or implanted follicularunits, and a monitor configured to display the same.

The system and method of the present invention is especially useful whenimplemented on, or integrated with, an automated system, for example, arobotic system comprising a robotic arm.

According to another aspect, a system and a method for controlling adirection of travel of a tool relative to a body surface is provided, inwhich the tool is caused to move or travel in an identified directionand operated to perform an action or procedure, for example, to harvestor implant follicular units in the direction of travel. The direction oftravel or, in some embodiments also a boundary, may be determined, forexample, based on a plurality of fiducials, which may comprise a set ofdistinctive fiducials. In some embodiments, the tool is operated totravel in a direction other than the direction of travel when anotherfiducial is within a predetermined distance from the tool. The directionother than the direction of travel may be substantially opposite thedirection of travel or may be substantially orthogonal to the directionof travel. A change of direction of travel may be substantiallyautomated based, for example, at least in part on the location of thefiducials.

According to yet another aspect, a system and method of operating a toolto harvest or implant hair grafts is provided, in which at least oneimage of a body surface is processed to divide the image into multiplerows and a tool may be operated to harvest or implant at least onefollicular unit in a first row. A determination may be made whether anumber of harvested or implanted follicular units in the first row iswithin a range of a desired number of harvested or implanted follicularunits for the first row. If the number of harvested or implantedfollicular units in the first row is within the range of the desirednumber of harvested or implanted follicular units for the first row, thetool may be moved to a subsequent row. However, if the number ofharvested or implanted follicular units in the first row is less than alower threshold value of the range of the desired number of harvested orimplanted follicular units for the first row, harvesting or implantingthe at least one additional follicular unit in the first row may becontinued.

According to still another aspect, a system and method of operating atool to harvest or implant hair grafts is provided, in which one or moreimages of a body surface are processed to determine locations of aplurality of distinctive fiducials appearing in the one or more images.The one or more images may be divided into multiple rows and a tool maybe operated to harvest or implant at least one follicular unit in afirst row at a first location. A direction of travel of the toolrelative to a body surface may be identified based on the first locationand the locations of at least some of the plurality of distinctivefiducials. The tool may be caused to travel in the identified directionof travel. A determination may be made whether a number of harvested orimplanted follicular units in the first row is within a range of adesired number of harvested or implanted follicular units for the firstrow. If the number of harvested or implanted follicular units in thefirst row is within the range of the desired number of harvested orimplanted follicular units for the first row, the tool may be moved to asubsequent row. However, if the number of harvested or implantedfollicular units in the first row is less than a lower threshold valueof the range of the desired number of harvested or implanted follicularunits for the first row, harvesting or implanting the at least oneadditional follicular unit in the first row may be continued. The toolmay then be operated to harvest or implant a second follicular unit at asecond location on the body surface in the direction of travel.

According to a further aspect, a processor is provided comprising a setof instructions for executing operations, the set of instructionsincluding instructions for processing one or more images of a bodysurface to determine locations of a plurality of distinctive referencesappearing in the one or more images; operating a tool to perform aprocedure or operation, for example, to harvest or implant a firstfollicular unit, at a first location; identifying a direction of travelof the tool relative to a body surface based on the first location andon the locations of at least one of the plurality of the distinctivereferences; causing the tool to travel in the identified direction oftravel; and operating the tool to perform an action or operation, forexample, to harvest or implant a second follicular unit, at a secondlocation on the body surface in the direction of travel. Theinstructions may further comprise utilizing at least one of theplurality of distinctive references to define a boundary, and operatingthe tool to perform the procedure or operation within the boundary.

According to yet another aspect, a method for defining a region ofoperation of a tool during a procedure or operation, for example hairtransplantation, is provided. The method may comprise selecting afiducial in an image of the body surface and moving, for example, animage acquisition device such that the fiducial is substantially at areference point in the field of view of the camera. A location of thefiducial is determined in a frame of reference of the body surface. Asubsequent fiducial may be selected, the subsequent fiducial being aclosest to the fiducial for which the location has been identified. Themethod further comprises moving the image acquisition device such thatthe subsequent fiducial is substantially at the reference point in thefield of view of the camera and determining a location of the subsequentfiducial with respect to the initial fiducial. The steps of selectingthe subsequent fiducial and determining the location of the subsequentfiducial may be substantially automatically repeated for a set offiducials that define a boundary of an area for performing a procedureor operation, for example for harvesting or implanting follicular unit.

According to a further aspect, a system and method for defining anexclusion region or zone is provided and information about the exclusionregion is used in determining the next procedure or operation location,for example, for determining the next harvesting or implantationlocation. The exclusion region is the region within which is notdesirable to perform a procedure or operation, for example, the regionfrom which harvesting follicular units or into which implantation of thefollicular units is not desirable. In some embodiments, the exclusionzone may be defined as a closed polygon, for example, a polygon ofsubstantially tear-drop shape on a surface of the body, or for example adonor area, such as scalp.

In various aspects, systems and methods may be provided that furtherenhance a selection of follicular unit harvesting or implanting sites.In various cases, such selection may be made using, for example, alowest and closest method, overlap-based methods, position-basedmethods, pattern-based methods (for example, a triangular pattern basedmethod), and/or a combination of these methods. In various combinationsof the methods, the output selection from one of the methods may be usedas an input for one of the other methods to select one of the bestcandidate sites.

Other and further objects and advantages of the invention will becomeapparent from the following detailed description when read in view ofthe accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

It should be noted that the drawings are not to scale and are intendedonly as an aid in conjunction with the explanations in the followingdetailed description. In the drawings, identical reference numbersidentify similar elements or acts. The sizes and relative positions ofelements in the drawings are not necessarily drawn to scale. Forexample, the shapes of various elements and angles are not drawn toscale, and some of these elements are arbitrarily enlarged andpositioned to improve drawing legibility. Further, the particular shapesof the elements as drawn, are not intended to convey any informationregarding the actual shape of the particular elements, and have beensolely selected for ease of recognition in the drawings. Features andadvantages of the present invention will become appreciated as the samebecome better understood with reference to the specification, claims,and appended drawings wherein:

FIG. 1 is a block diagram illustrating a general methodology of anexample of an embodiment according to one aspect of the invention.

FIG. 2 is a schematic representation of an example of a robotic systemthat could be implemented in various embodiments of the invention.

FIG. 3 is an example of a skin tensioner including fiducials which couldbe utilized in various embodiments of the invention.

FIGS. 4(a)-(f) show various examples of implementations of themethodology according to an embodiment of the invention.

FIGS. 4(g)-(h) demonstrate an example of an embodiment according to “anexclusion zone” methodology.

FIGS. 5(a)-(g) show an example of identifying and recording fiducials ascould be implemented in an example of the embodiment of the invention.

FIGS. 6(a)-(f) show another example of implementation of the methodologyaccording to an embodiment of the invention.

FIGS. 7(a)-(f) illustrate examples of various selection criteriaaccording to various embodiments of the invention.

FIGS. 8(a)-(f) are schematic representations illustrating an example ofthe use of satellite sites in the provision of exclusion zones.

FIGS. 9(a) and (b) illustrate the difference between using and not usingthe satellite exclusion zone methodology.

FIG. 10 is an example of a schematic representation of a reservedharvesting region.

FIG. 11 is a schematic representation of an example of a grid that maybe displayed on a monitor.

FIGS. 12(a)-12(h) illustrate a number of examples of different methodsfor selecting follicular unit harvesting or implantation sites that maybe used with various embodiments of the present disclosure.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following Detailed Description, reference is made to theaccompanying drawings that show by way of illustration some examples ofembodiments in which the invention may be practiced. In this regard,directional terminology, such as “right”, “left”, “upwards”,“downwards”, “vertical”, “horizontal” etc., are used with reference tothe orientation of the Figure(s) being described. Because components orembodiments of the present invention can be positioned or operated in anumber of different orientations, the directional terminology is usedfor purposes of illustration and is in no way limiting. It is to beunderstood that other embodiments may be utilized and structural orlogical changes may be made without departing from the scope of thepresent invention. The following description, therefore, is not to betaken in a limiting sense, and the scope of the present invention isdefined by the appended claims.

The term “tool”, as used herein refers to any number of tools or endeffectors that are capable of performing an action, procedure oroperation in various medical procedures or applications. For example,the tool may be a needle, a surgical scalpel, blades, various types offorceps, hemostats, surgical instruments, retractors, electrosurgicaltools, radio-frequency ablation tools, suturing devices, tattooplacement or removal tools, eye speculum, cannula, drills or lasers.With reference to hair transplantation procedures, a “tool” may comprisea “harvesting tool” or an “implantation tool”, and is capable ofdissecting, harvesting or implanting follicular units (“FUs”) from orinto a skin or body surface, for example, a scalp. Such tools may havemany different forms and configurations. In many embodiments, the toolcomprises a hollow tubular shaft and thus may be labeled, for example, acannula, a needle, or a punch. The distal end of such tools (forexample, punches, coring devices, cutting and/or trimming devices,needles), are typically sharpened, to various degrees, to penetratetissue and extract or implant the follicular unit. The terms“operatively connected,” “coupled,” or “mounted,” or “attached” as usedherein, means directly or indirectly coupled, attached, or mountedthrough one or more intervening components.

Embodiments of the methods of the present invention may be implementedusing computer software, firmware or hardware. Various programminglanguages and operating systems may be used to implement the presentinvention.

Hair transplantation procedures that are carried out using automated(including robotic) systems or computer-controlled systems have beendescribed, for example, in the Publication No. US 2007/0106306 commonlyowned by the assignee of the present application, which is incorporatedherein by reference. Robotics systems, such as robotic hairtransplantation systems generally require accurate positioning of a toolunder robotic control. When implementing a semi-automated or a fullyautomated procedure that requires precise control of the position, suchas hair transplantation, it is desirable to be able to maintain suchprecise control despite patient motion or temporary interruptions.According to one aspect disclosed herein, the present applicationprovides methodology for negating the effects of patient's movement orprocedure interruptions. For example, the described methodology avoidsfurther delays related to repositioning of a patient relative to a robotor an automated tool, and/or need for potential recalibration or a newtreatment plan to be configured.

According to the various embodiments described herein, a variety ofmethodologies and systems are provided which enable a tool toautomatically proceed from where it left off prior to an interruptionthat the procedure may be subject to, continuing its operation andessentially providing a seamless operational procedure. The systems andmethods described herein enable the tool to maintain its direction oftravel over the patient's body surface that it had despite patient'smovement or other interruptions, to recognize where it has previouslyharvested follicular units or implanted them, and continue to travel inthat general direction to harvest or implant further follicular units.The inventions described herein enable the system to operate in afully-automated fashion, if desired, without requiring relocation of thebase of the robotic system, relocation of the body surface, physicianassistance or human intervention. In addition, the present inventionprovides methodologies that enable a tool operated by an automatedsystem or under computer control to be operated to change its directionof travel when required, without necessarily requiring humanintervention, although a user could overwrite any automated movement ifdesired.

Although the various examples and embodiments described herein will usefollicular units (naturally occurring aggregates of 1 to 4 hairfollicles) or hair grafts for purposes of describing the various aspectof the invention, it should be apparent that the general understandingof the various concepts discussed can be applied more broadly to otherappropriate applications. It should be understood that although themethods described herein are especially suited for use with a roboticsystem for hair harvesting and/or implanting, they can be applied toother automated and/or computer-implemented applications. For example,devices, systems and methods described herein may be utilized in variousablation procedures (e.g. radiation-based), biopsy procedures, spinalprocedures, dermatological procedures (e.g., tattooing or tattooremovals, ophthalmic procedures, or treating various dermatologicalconditions, such as skin cancers). It should be noted that the examplesgiven herein are for the purposes of illustration and example only, thedescription as set forth is not intended to be exhaustive or limiting.

FIG. 1 is a block diagram illustrating an example of a generalmethodology employed by the present invention. At step 110 (which may bea preliminary step and it is shown in dotted line), one or more imagesof the body surface with one or more reference points, such as aplurality of fiducials, may be obtained, for example, using an imageacquisition device. That may be accomplished by any technique known inthe art. For example, in some embodiments an image acquisition devicemay be attached to a robotic arm, and the robotic arm with the attachedimage acquisition device may be positioned so that the harvesting orimplantation region is in focus for the cameras. In other embodiments,the image acquisition device may be still incorporated into theautomated (e.g., robotic) system but it does not have to be attached tothe robotic arm. Alternatively, in further embodiments, the imageacquisition device could be a device separate from the robotic system.As used in this application, a fiducial is an object that may act as areference, and may be identifiable in a field of view of an imagingdevice. Fiducials can take many forms, for example, a single artificialreference point that uniquely identifies both position and orientationmay be used as a fiducial. Take for example, a set of coordinate axesprinted on a surface. The origin, together with the directions of the Xand Y axes, can uniquely identify the surface position and orientation.In another example, a set of artificial reference points that eachuniquely specify a position can be used as fiducials. The combination ofthree or more such reference points can specify a unique frame ofreference specifying both position and orientation. An example would bespheres with different colors. One sphere uniquely specifies a positionin space, but not orientation. Two more spheres can be used to specifyboth position and orientation. In yet another example, natural featuresof a surface that have unique, recognizable patterns may be used asfiducials.

With reference to hair harvesting or hair transplantation or otherprocedures that could be performed on a body surface (including variouslayers of skin, face and its various parts, such as eyes, nose,eyebrows, etc.), natural physical features or anatomical landmarkspresent on the skin or other body surface that have unique, recognizablepatterns (e.g., follicular units or hairs, moles, scars, freckles,wrinkles, bumps or depressions on the body surface, eye balls, earcanals) may be used as fiducials. In the case of natural physicalfeatures or anatomical landmarks, these may be distinctive from oneanother based on their distinctive physical attributes (including butnot limited to size, color, shape, number, height from the body surfaceetc.) or their relative distance from another distinctive feature. Forexample, working on the surface of a head, the random dot pattern of theentry locations of hairs on the surface of the head is sufficientlyunique that a group of them can be used to unambiguously identifyposition and/or orientation. A pattern-matching algorithm can be used torecognize the hair pattern in subsequent images. In some embodiments,the fiducials may also be objects placed on or affixed to the patient'sskin, sometimes called external fiducials. In the embodiments whereexternal fiducials are used, they may be placed or affixed eitherdirectly to the skin surface in the hair donor or hair recipient area,or alternatively they may be placed on some device or instrument whichis in turn affixed to the body, for example, a skin tensioner used inthe hair transplantation procedures, as explained in more details inreference to the examples of FIGS. 3 and 4.

At step 115, a processor or an image processor, an example of which isdescribed later in reference to FIG. 2, processes and records anidentity and a location of each of the fiducials in a frame of referenceof an image acquisition device (e.g., in a camera field of view). Suchinitial recording of fiducials could be referred to as “fiducialregistration.” The fiducials could be recorded in various coordinatesystems, for example, in a fixed “world” coordinate system. In theexample of FIGS. 4(a)-4(f), the fiducials are described as recorded in acoordinate system fixed to the camera. In situations in which an imageacquired by the image acquisition device includes only a subset of thefiducials such that images of additional fiducials are needed, step 120provides for acquiring additional images as needed, for example,including other subsets of the fiducials, until all fiducials have beenidentified. (This aspect will be described in greater detail withrespect to FIG. 5). In an optional step 125 (shown in dotted line),based on the location of the each of the plurality of fiducials, aboundary of an area, such an area within which hair grafts or follicularunits are intended to be harvested from or implanted into, may bedetermined. The boundaries may be determined automatically, for example,by drawing lines between various fiducials. The boundaries may be alsoadjusted to eliminate certain portions of the bound area whereharvesting or implantation is difficult, as explained and described infurther detail in reference to FIG. 3. In order to accommodate forpatient motion, temporary interruptions, and any other incident that maycause a shift in location of the fiducials in the camera referenceframe, as often as required (as may be determined by the user), updatedimages of the body surface are acquired, the images containing an imageof the plurality of fiducials or a subset thereof. Due to patientmotion, or another such temporary interruption, the locations of thefiducials in these updated images may be in a revised location withrespect to the frame of reference of the image acquisition device. Theprocessor in step 130 processes the revised location of each of theplurality of fiducials in the frame of reference of the imageacquisition device, the revised locations of each of the plurality offiducials which may be different from the locations previouslyprocessed. Having acquired the revised locations of the fiducials, andwith the knowledge of the original locations of the fiducials, an offsetfor at least some or all of the fiducial locations may be determined instep 130. Based on this offset information, the processor also in step130 may process revised locations for each of the locations of interest,such as locations from which follicular units have already beenharvested (if harvesting has already started in a region of interestwithin the boundary) or into which follicular units have already beenimplanted (if such implanting has been started). Optionally, step 130may also comprise determining the revised boundary, for example, of theharvesting/implanting area based on the revised locations of thefiducials. However, it is not necessary in some embodiments to determinethe whole revised boundary as this information may be automaticallyascertained simply based on the offset of the minimum number of thefiducials. In reference to the example of hair transplantation, havingdetermined the offsets, and with the knowledge of the locations of thefollicular units that have been harvested or implanted (if any) withrespect of the fiducials, it is possible in step 135 to determine orselect a location from where the next hair follicle is to be harvestedsuch that hair follicles are not taken from an already harvestedlocation, or determine a location into which the next hair follicle isto be implanted such that hair follicles are not implanted intolocations into which hair follicles have already been implanted. Suchselection may be made using a processor programmed to perform theabove-described step, such as a processor described in reference to FIG.2. In step 140, for example, a tool may be moved to the selectedprocedure site, and in some embodiments, the procedure may be performedat the selected procedure site: for example, a hair graft or follicularunit may be harvested from or implanted into the selected location. Whenthe next hair follicle is harvested or implanted, the location fromwhere it has been harvested from, or implanted into, may be registeredor recorded by the processor in step 145. This registration may includeinformation on the location of the harvest or implant with respect to atleast one of the plurality of fiducials, or the determined boundary.Optionally, in step 150, the method may comprise creating and displayinga virtual representation on the image of the location from which thefollicular unit has been harvested (or at least dissected from thesurrounding tissue for further removal using forceps or vacuum), or thelocation into which a follicular unit has been implanted. Such visualrepresentation, for example, on a monitor (e.g. a computer screen) isespecially beneficial for the user to easily and quickly identifylocations where hair grafts have been dissected or harvested, and alsoto differentiate between the previously existing follicular units andthe newly implanted ones. The visual representations of step 150 may beimplemented by using different colors, shapes or other appropriatedifferentiating features. In step 155 the processor determines, based onthe information it has recorded with respect to the area and thelocations of the follicular units that have been harvested or implanted,if follicular units have been harvested from all desired sites, or iffollicular units have been implanted into all desired sites. In theevent that all follicular units have been harvested or implanted, theprocessor may communicate this information, for example, to the imageacquisition device. In addition, the processor may communicate thisinformation to the user, typically providing an indication to the user(via the monitor, voice command, or any other appropriate technique),for example, that step 110 may begin again at a new donor or recipientregion. In the event there are still follicular units to harvest orimplant, the processor continues to repeat steps 130-155 until alldesired follicular unit are harvested or implanted. For example, updatedimages with the updated fiducial information are processes, offsetsdetermined, the next harvest site or implant site is selected, etc. Inthis manner, a methodology is provided to enable hair follicles tocontinue to be harvested from or implanted into a body surface in acontinuous and automatic fashion despite potential patient movements andinterruptions. The tool is able to be moved to each new harvesting orimplantation location with respect to fiducials, the fiducials providinga mechanism of recognizing the location of the harvesting/implantingarea on the body surface, despite movement of the patient, or the imageacquisition device.

Referring first to FIG. 2, an example of a system that may be used withthe present invention is schematically shown. FIG. 2 is a schematicperspective view of an example of a robotic system 200 for hairharvesting (and/or implantation). The system 200 includes a robotic arm205 to which is coupled a tool 210. Various motors and other movementdevices may be incorporated to enable fine movements of an operating tipof the tool 210 in multiple directions. The robotic system 200 furtherincludes at least one image acquisition device 215, which is describedin more detail below. The image acquisition device may be mounted in afixed position, or it may be coupled (directly or indirectly) to arobotic arm 205 or other controllable motion device. The operating tipof the tool 210 is shown positioned over a body surface 220, in thiscase a part of the patient scalp having hair follicles thereon. In someembodiments, an image acquisition device may be provided separately andnot included in the system. In those embodiments, an interface may beprovided that allows various other components or modules of the system,such as image processing component, to interact with the separate imageacquisition device.

A processor 225 of FIG. 2, may comprise an image processor 230 forprocessing images obtained from the image acquisition device 215. Theimage processor 230 may be a separate device or it may be incorporatedas a part of the processor 225. The processor 225 may also instruct thevarious movement devices of the robotic arm 205, including the tool 210that may be operatively connected to the robotic arm. The processor 225may act, for example, through a controller 235 as schematically shown inFIG. 2. The controller 235 may be operatively coupled to the robotic armand configured to control the motion of the robotic arm, including themotion based on the images or data acquired by the image acquisitiondevice. Alternatively, controller 235 may be incorporated as a part ofthe processor 225, so that all processing and controls of all movementsof all the tools, the robotic arm and any other moveable parts of theassembly, including those based on the images or data acquired by theimage acquisition device, are concentrated in one place. The system 200may further comprise a monitor 240, keyboard 245, and mouse 250. Amagnified image of the body surface 220 can be seen on the monitor 240.In addition, the system 200 may comprise other tools, devices andcomponents, for example, those useful in harvesting, and/or implantationof the hair follicles, or in hair treatment planning. The system furthercomprises an interface adapted to receive an image data, various partsof the system allow an operator to monitor conditions and provideinstructions, as needed. The processor 225 may interact with the imagingdevice 215 via the interface (not shown). The interface may includehardware ports, cables, leads, and other data transmission means, or itmay comprise a computer program.

Some non-limiting examples of an image acquisition device 215 shown inFIG. 2 include one or more cameras, such as any commercially availablecameras. Of course, various image capture devices (or imaging devices)could be used with any of the embodiments of the systems and methodsdescribed herein. For example, the imaging device may be one or morecameras, such as any commercially available cameras. While stereo ormulti-view imaging devices are very useful in the present invention, itis not necessary to employ such geometries or configurations, and thepresent invention is not so limited. Likewise, although it is preferredthat the image acquisition device be a digital device, it is notnecessary. For example, the image acquisition device could be an analogTV camera that acquires an initial image which is then processed into adigital image (for example, via an analog-to-digital device like acommercial-off-the-shelf frame grabber) for further use in the method ofthe present invention. The image acquisition device may be coupled to aprocessing system, shown incorporated in the processor 225 in FIG. 2, tocontrol the imaging operation and process image data. The processor foruse in the present invention may comprise any suitable device programmedand configured to perform various methods described in detail in thepresent application, including methods directed to automated movement ofthe hair harvesting/implantation tool to maintain or change a desireddirection of travel within a hair donor or hair recipient area; ormethods directed in reference to FIGS. 4-7. For example, the processorfor use in the present invention may be a processor comprising a set ofinstructions for executing operations, the set of instructions includinginstructions for processing one or more images of a body surface todetermine locations of a plurality of distinctive fiducials appearing inthe one or more images, (in some embodiments, the plurality of thedistinctive fiducials may define a boundary); moving the tool to, andoperating a tool to harvest or implant a first follicular unit at afirst location; identifying a direction of travel of the tool relativeto a body surface based on the first location and on the locations of atleast one of the plurality of the distinctive fiducials; causing thetool to travel in the identified direction of travel; and operating thetool to harvest or implant a second follicular unit at a second locationon the body surface in the direction of travel. It will be understood bythose of ordinary skill in the art that the image processor for use withthe present invention is programmed and configured to perform variousknown image processing techniques, for example, segmentation, edgedetection, object recognition and selection. These techniques aregenerally known and do not need to be separately described here.

By way of example, and not limitation, a suitable processor or imageprocessor may be a digital processing system which includes one or moreprocessors or other type of device. For example, a processor (imageprocessor) may be a controller or any type of personal computer (“PC”).Alternatively, the processor (image processor) may comprise anApplication Specific Integrated Circuit (ASIC) or Field ProgrammableGate Array (FPGA). The processor may also include memory, storagedevices, and other components generally known in the art and, therefore,they do not need to be described in detail here. The above-describedprocessor could be used in conjunction with various partially automatedand fully automated (including robotic) hair transplantation andtreatment systems and devices, including but not limited to systems forhair harvesting, or hair transplantation.

In order to better understand how the method of FIG. 1 can beimplemented utilizing a system such as that illustrated in FIG. 2, anexample of arrangement of external fiducials is described in referenceto FIG. 3. While performing hair harvesting, especially when using anautomated system, such as a robotic system, it is often desirable tostretch the skin around the area of location of the follicular unit tobe harvested. This could be done using a skin tensioner device. FIG. 3illustrates an example in which a set of unique or distinctive (meaningthat they are distinguishable or different from each other) fiducialsare either formed on or affixed to a frame of a skin tensioner 300 thatcould be used in the hair transplant procedure to tension a skin surfacefrom which hair follicles are harvested. Such skin tensioner could alsobe used in some embodiments during hair implantation if tensioning theskin surface is desired, in other embodiments fiducials could be placeddirectly on the skin surface during hair graft implantation. Moreover,in some embodiments natural features (e.g., moles, scars, etc.) could beused as fiducials. While the following discussion of the fiducials andtheir use will be described in reference to FIG. 3 and the skintensioner, people of ordinary skill in the art would understand thatthis description may be adjusted and it is intended that the sameprinciples should be applied to the fiducials placed, for example,directly on the skin or on the device other than the skin tensioner, aswell as to the natural fiducials previously identified (e.g., follicularunits or other physical landmarks). Therefore, such alternativeimplementations are within the scope of the invention. The skintensioner 300 may comprise a flexible frame 305 that lies generally in aplane and shown to comprise a single element, typically molded materialand is configured such that it may be compressed inward from a relaxedposition. The flexible frame 305, in the example illustrated, includesfour side sections 310. The four side sections 310 are shown linear andarranged substantially in a square, although they may be arcuate andotherwise arranged in various geometrical patterns. Each side section310 features a plurality of perforations 315 for receiving barbs ormicrobarbs (not shown), which are small elements that project from skintensioner to the skin below to puncture or form a depression in the skinsurface, serving as a primary means of ensuring good grip between theskin tensioner and the skin. The illustrated fiducials 320 may comprise,for example, a set of circular fiducials 325, each circular fiducial 325being distinguishable from any other circular fiducial, and a set ofsquare fiducials 330, each square fiducial 330 being distinguishablefrom any other square fiducial. For example, as illustrated, thefiducials may comprise a single feature, for example a dot, and eachfiducial may be distinguishable from the others by the size of thefeature (e.g., the dot). Alternatively the fiducials may comprise afeature (such as a dot) that may be of the same (or different) size oneach fiducial, but the fiducials may be further distinguishable from theone another, for example, by the number of the features (such as dots)that it has on it. In further alternative embodiments, each fiducial maycomprise a different feature or features. The fiducials 320 can be ofany shape or configuration, provided the imaging system is capable ofdistinguishing one fiducial from another. In the illustrated embodimentthe imaging recognition software could, for example, recognize whetherthe fiducial it had identified was a circular fiducial 325 or a squarefiducial 330 by determining the ratio of the square of the perimeter ofthe fiducial to the area of the fiducial. For example, this ratio for acircular fiducial 325 is around 12.5, and that for a square fiducial 330is 16. Hence, having made this determination, the processor containingimage processing software would be able to distinguish the set of 6circular fiducials 325 from the set of 4 square fiducials 330,illustrated as an example in FIG. 3. In addition, a similar capabilityin image processing could enable the processor to determine that theratio of the area of the single smaller dot 335 on the circular fiducial325 to the area of the circular fiducial itself, was smaller than theratio of a larger single dot 340 on the circular fiducial 325. Henceuniquely identifiable fiducials 320, such as those illustrated byexample in FIG. 3, could be used to implement the methods of the presentapplication. The fiducials 320 may be placed at a known distance awayfrom the inner edges that form a central opening 350 of the skintensioner. In the example of FIG. 3, this known distance is illustratedas a distance 355 and it is shown as a distance from an inner edge ofthe tensioner to a point 345 on a fiducial where the point 345 is theclosest to the central opening 350 of the skin tensioner 320.Alternatively, a distance 355 may be measured from the inner edge of theskin tensioner to a center of the relevant fiducial. For example, insome embodiments the row of fiducials is placed such that the distance355 may range between 1 mm and 10 mm (and in some embodiments mayfurther range between 2 mm and 4 mm) from the inner edges of thetensioning device 300. In other embodiments, depending on theapplication, this distance 355 may have widely different ranges.Fiducials may be used to bound an area, for example, for harvesting, andmay need to have a known relationship to that area. The processor maycomprise an algorithm which detects the fiducials on each side of theskin tensioner (or on the skin or other alternative surface), fits linesto the four (in the example of FIG. 3) rows of fiducials and computes aquadrilateral whose corners are formed by the intersection of thebest-fit plane containing all fiducials with planes passing through eachline and normal to the all-fiducial plane. Based on the above and on theknown distance from the near edge of the skin tensioner to thefiducials, the processor is consequently able to identify and calculatethe area or central opening 350 bounded by the four side sections of theskin tensioner, which could represent the area within which it isdesired that hair follicles be harvested from (or implanted into).

Utilization of a skin tensioner 300 to host the fiducials 320, mayrequire other factors be taken into consideration when identifying theactual bound area where follicular units will be harvested or implanted.One such factor is that the skin tensioner itself has a depth or heightassociated with it, that is, it does not typically lie flush with thepatient's body surface, but is raised above the body surface to acertain degree. It will also be appreciated that the angle at which thefollicular units extend from the patient's body surface varies. To thisend, there may be situations in which although there may be a follicularunit that is close to the inner edges of the skin tensioner 300, due tothe depth/height of the skin tensioner and/or the angle at which thefollicular unit emerges from the skin, the tool that will be placedinside the central opening 350 of the tensioner may not be able to beoriented properly relative to the follicular unit without interferingwith the inner edges of the tensioner that define the opening 350.Therefore, a successful harvesting of the follicular unit could not beattempted. For this reason, in addition to using the information of aknown distance from the fiducials to the inner edges defining thecentral opening 350, the processor may be also configured to take intoconsideration, for example, a depth or height of the inner edge of thetensioner, and/or an angle and dimensions of the tool/punch when itorients relative to a hair graft to harvest it (or relative to a desiredorientation of the hair graft to be implanted). When these distances,angles and other relevant parameters are taken into account, theprocessor may determine, using a straightforward trigonometriccalculation, a revised boundary 360. This revised boundary 360 providesa predetermined distance from the fiducials that the tool may safelyapproach, without encountering the physical inner boundaries of thetensioning device itself, encountering issues arising from one or bothof the depth/height associated with the tensioning device, encounteringimage processing issues arising from the shadow cast by the tensioningdevice on the body surface, and/or the angle of approach of the tool.Alternatively, a calculation may be performed on each hair in the fieldof view in order to allow selection of only those hairs that arereachable without such interference from the tensioning device or issuesarising from inadequate tool dimensions. This selection may be done by auser based on user-specified criteria input via, for example, akeyboard, selected via the mouse, or selection may be provided by anautomated algorithm, to harvest or implant the next follicular unitaccordingly. Having considered and accounted for all these variations,the location of the fiducials can be used to calculate whether the hairharvesting or implantation tool will clear the tensioner during thetransplantation procedure.

Additional input criteria relating to the parameters of the proceduremay also be selected at this time, allowing for automation of theprocedure. Similar to that described above, these parameters may beinput via a keyboard or a remote input device, or selected via themouse, or selection may be provided by drop-down menus of an automatedalgorithm or similar such means. In this manner, the user may select, inreference to hair harvesting or implantation, for example, that theminimum distance from any previous harvest site be at least 2.0 mm, theminimum distance from a previously skipped harvest site be, for example,at least 0.5 mm, similarly, tensioner offset distance from each side maybe selected as well, or the type of follicular unit to be harvested (F2,F3, or F4, for example), or any other such parameter(s). With respect toother procedures, the appropriate parameters for such specific proceduremay be selected in a similar manner. For example, in a laser tattooremoval application, the user may select the angle of the laser to thebody surface and/or the distance of the laser with respect to the skin.If instead of a tensioner some other boundary setting device is used,various distances as described above (e.g., offset on each side, etc.)may be selected by the user.

The present invention utilizes of a set of identifying fiducials such asthose described above, to facilitate automation of a robotic system,such as follicular unit harvesting or implanting system. In someembodiments one or more of the fiducials are distinguishable fromothers, in others, all of the fiducials are distinguishable from eachother. The fiducials serve as objects, or reference marks in a field ofview of an image acquisition device. These identifying fiducials whenviewed in an image can be recognized in the image, and may beindividually recognizable from each other in subsequent images.Fiducials may be physically identified by a 1-D bar code, a 2-D datamatrix code, known markings such as alphanumeric characters, a series ofdots, a series of bars, or any other type of unique identifier or customscheme. As mentioned above, the perimeter-to-area ratio, the ratio ofarea of the internal features to the outside features, and the number ofinternal features may be combined to ensure that a unique identifier canbe determined for each fiducial.

FIGS. 4(a)-4(f) illustrate how the steps of FIG. 1 can be implementedutilizing the system of FIG. 2 and fiducials, for example, similar tothose as illustrated in FIG. 3. In FIG. 4(a), a tensioning device 400 isillustrated. In order to more easily explain the various methods of thecurrent invention, rather than utilize fiducials such as thoseillustrated in FIG. 3, the discussion will utilize fiducials 405 whichare illustrated as a series of alphabetic characters, A to F, along thevertical side sections of the skin tensioner, and a set of numericcharacters, 1 to 8, along the horizontal side sections of the skintensioner. The fiducials generally may be placed in arbitrary positionsrelative to a working area. As mentioned above, the fiducials are placedsuch that a known feature of the fiducial, for example the center ofeach fiducial, or a known boundary of the fiducial is at a knowndistance from inner bounding edge of the skin tensioner. For example,the fiducials may comprise circular shaped adhesive labels that affix tothe tensioning device, the edges of the circular shaped adhesive labelsbeing such that the size of the label is less than the size of thevertical and horizontal structures of the tensioning device to which itis attached, so that when placed on the tensioning device, the fiducialsthemselves may be located, for example, in the region of 2 mm or so fromthe inner boundary of the tensioning device. It is desirable to positionfiducials (whether it is on the tensioner, or on a skin itself) orlocate natural fiducials, such as follicular units, such that therelative position of the fiducials do not change, or if they do, they donot change significantly during the procedure. The system, in particularthe processor, can process images acquired by the image acquisitiondevice to detect substantial relative motion, for example motion in theregion of 1 mm in a field of view in the region of 50 mm, that may becaused, for example, by misidentification or detachment of a fiducial(if it is an externally placed fiducial), and report such an error sothat measures may be taken if necessary to compensate or correct for theerror. Of course, the surface on which the fiducials are located is freeto move (e.g. patient moving his/her head or getting up). Thetranslation and rotation of the surface with the fiducials due tomotions can be computed any time when at least three non-collinearfiducials are visible. When the shifted and/or rotated fiduciallocations are detected (for example, by an imaging device and/or imageprocessing software or hardware which may form a part of an imagingsystem), the procedure is able to continue at the next harvest site. Iffewer than three non-collinear fiducials are visible, another attempt toimage and register fiducials is made so that three or more fiducialsbecome visible, as described in more detail in reference to FIGS.5(a)-5(g).

According to the methodology of an embodiment of the invention, and withreference to FIG. 4(a) the location and optionally the orientation ofeach of the fiducials 405 may be identified, registered, andelectronically saved via the image processor. In addition, the user mayalso specify via an input device such as the keyboard or the mouse,information pertaining to the physical parameters of the tensioningdevice 400 (if such tensioning device is used) and the tool, informationsuch as the height of the tensioning device 400 relative to the bodysurface and the diameter of the tool for example. Based on thisinformation, the processor may, optionally, determine the location andorientation of a revised boundary 410. To aid in the understanding, itwill be assumed that the fiducials 405 are recorded in the coordinatesystem of the image acquisition device, which as indicated above is onthe robotic arm, though they may be recorded in any appropriatereference frame.

Having registered the location and possibly the orientation (whenapplicable) of each of the fiducials 405, the image processor identifiesthe location and optionally the orientation, of one or more hairharvesting (or implantation) sites 415, and may register andelectronically save such identified information. Optionally, if therevised boundary 410 has been determined, the image processor identifiesthe location and possibly the orientation of one or more hair harvesting(or implantation) sites 415 within the revised boundary 410. Theinformation about location and orientation of the harvesting (orimplantation) site is registered and stored with respect to the locationand orientation of the fiducials 405. This enables monitoring andcontrol, for example, of the spacing between hairs to avoidunderharvesting (when harvest density is too low) and overharvesting(when harvest density is too high). Optimal density can be maintainedonly if the system, such as the robotic system is able to maintain itsknowledge of the harvest (implant) area, and use the full area availablefor harvesting or implanting. When the fiducials are used to define theboundaries, for example, of the harvest area, harvesting may beautomatically performed as close as desired to that boundary. Theharvesting mechanism can turn automatically to start a new row when theboundary is approached, and can stop automatically when the full areabounded by the fiducials has been harvested. Automation of the hairharvesting (or hair implantation) procedure is facilitated bymaintaining harvest direction and row-to-row spacing despite patientmotion as discussed below.

FIGS. 4(a)-4(f) will be discussed, as an example, in reference to thehair harvesting, and assuming that revised boundaries 410 are utilized.However, it should be understood that this description applies and couldbe adjusted accordingly in reference to hair implantation, and toeliminate the determination of revised boundaries 410. As illustrated inFIG. 4(a) the harvesting tool is operated to initiate the harvestingprocedure from the bottom left hand corner of the quadrilaterally shapedbound area, bounded by the revised boundary 410. During the hairtransplantation process, often various fluids, including for example,blood and saline will be present on the body surface. It was discoveredthat it is advisable, especially in the computer-implemented or robotichair transplantation procedures, that the harvesting or implantationprocess begin from the bottom of the frame, whether it be in the rightor left corner. This way any appearing blood or other fluid will tend toflow downwards, and therefore, will less likely compromise the image ofthe potential subsequent hair harvesting or implantation sites, thusoptimizing any image processing that may be implemented. The harvestingtool may be moved to the initial or first harvesting location, such aslocation 415 in FIG. 4(a), directly or indirectly by the physician (forexample, the physician may click on the image to identify the desiredharvesting location), or the processor may be configured or programmedto find this location itself, for example, based on the information ithas acquired on the fiducials, the processor may then provideinstructions to the control unit to move the tool accordingly.

In this particular instance, the tool has been instructed to move to thelocation approximately corresponding to the position C5, and the tool isoperated to harvest a follicular unit at the harvesting site 415. In oneembodiment according to the provided methodology, the processor maycreate a visual representation of the location on the site at which thefollicular unit has been harvested This representation may comprise acircular shape such as that illustrated, a cross, or any other suchvisualization. The visual representation or the marking of theharvesting (or implantation) site is beneficial to the user of thesystem, providing a visual image of where harvests (or implantations)have occurred. Moreover, in some embodiments, it may be desirable tohighlight the above-mentioned visual representation of the harvesting orimplantation site in a distinctive color. The tool is then controlled tomove in the direction represented by arrow 420, along the row defined byvirtual line C-F, substantially parallel to a horizontal side of therevised boundary 410. Although for convenience, the drawings of thisapplication show that follicular unit implantation or harvesting takesplace in straight rows and columns, it should be apparent to those inthe field that naturally grown follicular units do not grow in straightrows and columns, and needless to say it is not intended that thepresent invention be read is such a restrictive fashion. The column androw explanation has been used for ease of understanding only, andlocations at any reference location fall within the scope of theapplication.

As indicated in FIG. 4(b) the processor can be configured to operate thetool to harvest follicular units at predetermined locations, such as inthis example, by passing over, for example, any locations in row C-Fbetween fiducial locations 5 and 7 without harvesting a follicular unit,and making its subsequent or second harvest at the location 425 thatcould be defined as C7. It should be understood that the selectedharvesting location, for example, on the row C-F does not have to beexactly at the level of the location of the fiducials 405 (such asfiducial 7), but rather may be anywhere and at any distance from aparticular fiducial (e.g., between the level of fiducials 6 and 7). Whenthe processor determines that the tool is within a predetermineddistance from fiducial F and the end of this first harvesting row, orthat the revised boundary 410 has been reached, the processor providesinstructions to the control unit to cause the tool to move in adirection away from fiducial F to automatically increment to the nextharvesting row. In this particular case, as illustrated in FIG. 4(c),the tool is controlled such that it moves initially in an upwardlydirection 430, from F to E, and then in a direction denoted by arrow435, away from E, away from the revised boundary 410, and along thevirtual line EB. In this case, the tool is controlled to move to aharvesting site 440 located at approximately B-8, and operated toharvest a follicular unit at that location before moving on. Thisprocedure can continue without requiring intervention from the operatoror physician. It should be understood, however, that the operator mayintervene at any time to overwrite an automated movement and select adifferent follicular unit to be harvested, if desired or necessary. Thesystem is configured to direct a tool to move and operate, for example,at least in part based on the location of the fiducials. In this manner,the tool can be operated to turn automatically when the revised boundaryis approached and start the next row of the harvesting process, and tostop automatically when the area bound by the revised boundary 410 hashad all desired follicular units harvested.

According to another example of implementation according to theinventions described herein, FIG. 4(d) indicates a location 445 denotedby an “X”, at which it is intended that the next follicular unit beharvested. However, let us assume that for whatever reason, there is aninterruption, perhaps the patient moves, either temporarily leaving theoperation chair, or just shifting to get in a more comfortable position.Even though the patient has moved, the view seen by the camera which isin this instance located on the robotic arm, will be substantiallyunchanged in the global context, that is, the view relative to the chairwill be the same (assuming that the chair is not moved with respect tothe robot). However, the view with respect to the patient's body surfacemay be different. As indicated in FIG. 4(e), the patient's body surfacemay have moved such that the existing follicular units can be seen tohave moved both to the right, and upwardly, in the frame of view 450. Ifthe tool was moved to harvest the next follicular unit at a locationwith coordinates referenced with respect to the frame of view 450, itcan be seen that the follicular unit would be harvested from a location455 marked with the “X” which is not the original desired locationmarked as 445 in FIG. 4(d). As indicated in FIG. 4(e), this location 455is close to another follicular unit 460 that could be damaged byharvesting the location 455. This location 455 is also close to alocation from which another follicular unit has already been harvested,location 465. Moreover, this would not maintain the intended row-to-rowspacing of harvested follicular units, and does not maintain theintended spacing of the harvested follicular units from one another. Byregistering the intended harvesting site with respect to the fiducials405, the robotic system is able to avoid some or all of these errors,and additionally is able to continue the harvesting process withoutnecessarily requiring significant intervention of the physician to doso. The robotic system is configured to determine the location andorientation of each of the fiducials 405, and compare these new orrevised locations and orientations with the already saved information oneach of the distinctive fiducials 405. For example, in this particularcase, it will determine that the location of each of the fiducials 405has moved a certain distance towards the right hand side of the frame ofview, and a certain distance in an upwardly direction also. Usingfiducial tracking techniques that are known in the art, the system isable to determine how each of the initially identified fiducials hasbeen transformed in location and orientation, and determine thetransformation that needs to be applied to the location 455, to relocatethat same location 455 with respect to the fiducials 405. Havingacquired this information, the processor is then able, using knowntransformation techniques, to modify the location and orientationinformation of the intended implantation location 455 accordingly,applying the necessary transformation of the coordinates of thelocation, so that the tool can be operated to move to the correctharvesting site 470 (which correspond to the originally marked site445), as illustrated in FIG. 4(f). In this manner, the system isconfigured to operate the follicular unit harvesting tool to maintainits harvesting direction that is along the virtual row A-D, despitepatient movement. In addition, the system is configured to ensure thatharvesting does not occur at sites where harvesting has already takenplace, enabling site to site spacing and row-to-row spacing to bemaintained. To this end, the provision of visual image, for example, inthe form of circles centered about where harvesting has been performed,provides the user with a visual representation that confirms thatharvesting has occurred at the site to site spacing and row-to-rowspacing desired. Obviously, should the visual circular representationsnot correspond with the desired outcome, the user has an opportunity,based on the recognition of the deviation from expectation, to correctfor any errors that may be visualized. This correction can beimplemented at any moment that deviation from expectation has beendetected, and multiple corrections may be performed throughout anyprocedure. In this manner, the system is able to harvest follicularunits despite patient movement. It will be appreciated that although theabove has been described with respect to the harvesting process, themethodologies described above can be easily adapted to apply to theimplantation process, or other procedures.

According to another aspect of this application, harvesting andimplantation locations could be used to define “exclusion zones” aroundharvesting or implantation sites. For example, arbitrarily shapedfeatures or structures may be utilized to facilitate selection of thenext harvest or implant site, which may optionally be visuallyrepresented to the user. In one embodiment, the perimeter or an outlineof the arbitrarily shaped feature can be tailored to indicate anexclusion zone, that is an area within which selection of the nextpotential harvesting site or a potential implantation site should beavoided. A more detailed discussion of the exclusion zone as used in thepresent application is provided below.

To aid with the understanding of the exclusion zone, consider first asituation in which no harvesting or implanting is allowed when thedistance between the proposed site and any previous harvest site is lessthan a given radius, and the harvesting tool penetrates a body surfacesubstantially orthogonal to the body surface. In this situation, asimple circle (representing a simple exclusion zone) may be utilized tofacilitate selection of the subsequent hair harvesting or implantationsite, by creating the perimeter around a new potentialharvesting/implanting site. The perimeter of such circle will be largerthan the potential harvesting/implanting site to provide for anexclusion zone around the harvesting/implanting site, that is an area inwhich the tool should avoid harvesting/implanting a subsequentfollicular unit. For example, such harvesting/implanting should beavoided if, in addition to the potential harvesting/implantation site,there is also a location of already previously harvested/implanted sitewithin the perimeter of the circle, or perhaps within a predetermineddistance from the perimeter. The exclusion zone may be based on variouscriteria, including, for example, avoiding problems such as thepotential harvest/implant site coinciding, intersecting with, or comingtoo close to an already existing harvest/implant site, or merelydefining the minimum separation of follicular units that are to beharvested/implanted for medical or aesthetic reasons.

The above methodology works well if one assumes that the harvesting toolenters the body surface substantially orthogonal to the body surface.However, hairs do not generally grow orthogonal to the body surface, orat least the portion of the hair beneath body surface does not growsubstantially orthogonal to the body surface. Therefore, it is morelikely that a harvesting tool will be approaching a scalp at an angle.Assume that this angle is an acute angle. Due to the acute angle of theapproach, and the velocity of approach, the tool (such as a harvestingpunch) may tend to skive the skin, sliding a little further than perhapsoriginally intended, and enter the body surface slightly off-center fromthe intended harvesting site. As the punch enters the body surface, itis doing so at an angle, and therefore as it continues to penetrate intothe body tissue, it also does so at an angle. As the harvesting toolpenetrates the body surface, the distal end of the harvesting tool maynot only enter the body surface at a location that differs from theintended entry point (the intended harvesting site), but the distal endof the harvesting tool may also reach a location beneath the bodysurface that differs in the horizontal direction from the originallocation of entry on the body surface. It is therefore possible that onso doing, the distal end of the harvesting tool may coincide orintersect with an already harvested site, or a site that has alreadybeen implanted into. In this particular situation, relying on acircular-shaped feature may cause an unintentional overlap with anexisting harvesting or implantation location, and therefore, may createpotential problems. For example, harvesting a follicular unit that istoo close to a previous harvest site can cause the skin between the twoharvests to tear, resulting in excessive bleeding and scarring.

It is therefore an aspect of this disclosure to provide for an exclusionzone that is tailored to accommodate at least one or more of variousfactors, for example, with reference to hair transplantation, a minimumdistance between harvests, a minimum distance between implants, thediameter of the tool, the angle of approach of the tool, the directionand/or velocity of approach of the tool, or the depth of penetration ofthe tool. In reference to other medical procedures, an exclusion zonewill be tailored to the factors appropriate for such procedures. Such anexclusion zone may comprise any closed polygon-shaped feature, be itoval, elliptically-shaped, tear-drop shaped, or any arbitrarily shapedfeature configured to accommodate or take into consideration theexamples of the factors mentioned above. The parameters of the exclusionzone (its size, shape, and location) provide information that can beutilized by the processor in the selection of the next harvesting orimplantation site, to exclude harvesting or implanting into alreadyharvested or implanted regions, or too close to such regions, whetherthose regions be at the skin surface or below it. It also provides avisual indication to the user that appropriate selections of harvestingor implantation sites are being made by the automated hairtransplantation system.

According to one aspect, as an example, a method for defining anexclusion region of operation of a tool during hair transplantation isprovided. The method may comprise providing processing instructions thatcan cause an exclusion zone to be created around a potentialharvest/implant site, the exclusion zone may be based on at least one ofor more of a minimum distance between harvests, a minimum distancebetween implants, the diameter of the tool, the angle of approach of thetool, the direction and/or velocity of approach of the tool, or thedepth of penetration of the tool. The method further comprisesdetermining existence of any previous harvest or implant site that maylie within the exclusion zone, in addition to the proposedharvest/implant site. If a previous harvest or implant site lies withinthe exclusion zone, the proposed harvest or implant site is skipped, itis not harvested/implanted and the processor may select another proposedharvest or implant site, and check again. This selection process may becontinued until a site is selected that passes an exclusion zone test,for example, the test of having no previous harvest or implant siteswithin its exclusion zone.

FIGS. 4(g) and 4(h) illustrate an example of an embodiment using anexclusion region methodology. Turning first to FIG. 4(g), alreadyharvested locations 472 and 474 are identified with a circularrepresentations 476 and 478 indicating larger regions centered about theharvest locations 472 and 474. These circular representations 476 and478 provide a visual image of areas from which it is not desirable toharvest additional follicular units, as these would be too close toalready harvested follicular units. The processor may select a potentialharvesting location 480, which is outside the circular representations476 and 478. In addition, the processor creates an arbitrary shapedfeature 482, which in this instance is shaped as a tear-drop, around thepotential harvesting location 480. As can be seen in the figure, theshape of the feature 482 can be described as a circle around thepotential harvesting location 480, which has been extended, or stretchedin the direction 484, which is the direction in which the tool is bothtravelling and angled to harvest, thus forming a tear-drop shape (orexclusion zone). Having created this exclusion zone 482, the processordetermines whether there are any already harvested sites that fallwithin the exclusion zone 482 in addition to the potential harvestingsite 480. In this instance, the already existing harvesting site 472 canbe seen to fall within the exclusion zone of the tear-drop 482, and sothe processor will determine that this potential harvesting site 480 isnot a site from which a follicular unit should be harvested. Harvestinga follicular unit from this location 480, with the tool orientated atthe selected angle and in the direction 484, could possibly create aharvesting path that coincides or intersects with the already existingpath that was created at location 472. Therefore, the processor selectsan alternative potential harvesting site, for example that indicated inFIG. 4(h).

In FIG. 4(h), once again already harvested locations 472 and 474 areidentified with circular representations 476 and 478 indicating largerregions centered about the harvest locations 472 and 474, from which itis not desirable to harvest additional follicular units. The processorselects a potential harvesting location 486, which is outside thecircular representations 476 and 478. In addition, the processor createsan arbitrary shaped feature 490 (or exclusion zone) which once again isshaped as a tear-drop, around the potential harvesting location 486. Ascan be seen in the figure, the shape of the feature 490 can be describedas a circle around the potential harvesting location 486, which has beenextended, or stretched in the direction 488, which is the direction inwhich the tool is both travelling and angled to harvest, thus forming ashape that is tear-drop shaped. Having created this exclusion zone 490,the processor determines whether there are any already harvested sitesthat fall within the exclusion zone 490 in addition to the potentialharvesting site 486. In this instance, while there are follicular unitswithin the exclusion zone 490, for example, the harvesting site 492,none of them has been harvested yet. Therefore, a candidate or potentialharvesting site 486 is an acceptable candidate, and the processor maydetermine that this potential harvesting site 486 could be harvestedwithout intersecting with any already existing harvesting path.

The generation of visual representations that define exclusion zonesthat are centered, for example, about a harvesting site, may create animage that has numerous overlapping representations, and consequently animage that has numerous gaps formed between each of the distinctexclusion zones. This is illustrated in FIG. 8(a) in which alreadyexisting procedure sites, such as harvesting sites 805, 810 and 815,each have associated exclusion zones 820, 825 and 830 respectively,created as described hereinbefore. The exclusion zones 820, 825 and 830create a gap 835 as illustrated. Gaps such as these tend to create arather “unfriendly” visual representation for the user and theprocessor. An image with a vast number of these gaps can make itdifficult for the eye to easily identify or focus on the “larger” gaps,and may also consume additional processing time by the processor. Byavoiding the creation of these gaps, and in particular the relativelysmall gaps, a visual representation that is more pleasing to the eye canbe created, a more “friendly” visual representation for the processorand/or the user, one in which gaps are fewer and easier to identify.This is particularly beneficial, for example, in situations where manualfollicular unit selection is desired, situations in which for example,the user can manually select follicular units to be harvested that weremissed by an automatic selection algorithm or close to the skintensioner but still harvestable in the eyes of the user.

One way in which the gap 835 illustrated in FIG. 8(a) can be avoided, isby using an alternative or additional methodology which serves to fillthe gaps between the exclusion zones 820, 825 and 830. In oneembodiment, more than one existing follicular unit harvest site is usedto create a visual representation of the exclusion zone for thatparticular harvested follicular unit. For example, in one such methodfor creating a visual representation of the exclusion zone, the visualrepresentation is generated by using not only the newly harvestedfollicular unit, but by using information from its neighboring orsatellite sites, the sites of previously harvested follicular units thatare in close proximity to the newly harvested follicular unit.

FIG. 8(b) shows the two existing harvested follicular unit sites 805 and810, and a site 815 which represents a site of a newly harvestedfollicular unit. For ease of explanation, the exclusion zones 820, 825and 830 have been omitted. When processing the exclusion zone for thenewly harvested follicular unit site 815, the processor is configured todetermine whether or not the already existing follicular unit harvestingsite 805 is within, for example, a predetermined distance from it. Thispredetermined distance may be based, for example, on a multiple of theminimum harvest distance described above, the multiple being greaterthan one, and ranging, for example, from 1.5 to 2.5. For example thepredetermined distance may be less than or equal to at least twice theminimum harvest distance from the newly harvested follicular unit site815, that is a distance of, for example, 3.8 mm. If it is found that thealready existing follicular unit harvesting site 805, for example, isless than or equal to at least twice the minimum harvest distance awayfrom the newly harvested follicular unit site 815, the already existingfollicular unit harvesting site 805 may be considered to be a satellitesite with respect to the newly harvested follicular unit site 815. If,however, it is found that the already existing follicular unitharvesting site 805 is more than at least twice the minimum harvestdistance away from the newly harvested follicular unit site 815, thealready existing follicular unit harvesting site 805 may be considerednot to be a satellite site with respect to the newly harvestedfollicular unit site 815. In a similar manner, when processing theexclusion zone for the newly harvested follicular unit site 815, theprocessor is configured to determine whether or not the already existingfollicular unit harvesting site 810 is less than or equal to at leasttwice the minimum harvest distance from the newly harvested follicularunit site 815, that is a distance of, for example, 3.8 mm.

For ease of understanding, let us assume that both existing harvestedfollicular unit sites 805 and 810 are less than or equal to at least twotimes the minimum harvesting distance from the newly harvestedfollicular unit site 815. In this instance the processor creates aclosed loop profile, or a supplemental exclusion zone, based on thelocations of the newly harvested follicular unit site 815, and theexisting harvested follicular unit sites 805 and 810, forming atriangular shape 840 as illustrated in FIG. 8(b). The processor combinesor superimposes this closed loop triangular profile 840 (thesupplemental exclusion zone) onto the three circular exclusion zones820, 825 and 830, as illustrated in FIG. 8(c) to form a visualpresentation of the modified exclusion zone 845, as illustrated in FIG.8(d) that no longer incorporates the relatively small gap 835.

FIG. 8(e) illustrates an example in which a newly harvested follicularunit site 850 is introduced, sites 805, 810 and 815 all being alreadyexisting harvested follicular unit sites. If one assumes that theexisting harvested follicular unit sites 805, 810 and 815 are less thanor equal to a certain minimum harvesting distance, for example, at leasttwo times the minimum harvesting distance, from newly harvestedfollicular unit site 850, they will all be considered satellite sites tonewly harvested follicular unit site 850. The processor in this instanceis configured to create a closed loop profile (supplemental exclusionzone), based on the locations of the newly harvested follicular unitsite 850, and the existing harvested follicular unit sites 805, 810 and815, forming a polygon with indices 805, 810, 815 and 850. However if,for example, it is determined that only existing harvested follicularunit sites 810 and 815 are less than or equal to at least two times theminimum harvesting distance from newly harvested follicular unit site850, and existing follicular unit harvesting site 805 is more than atleast two times the minimum harvesting distance from the newly harvestedfollicular unit site 850, only existing harvested follicular unit sites810 and 815 will be considered satellite sites for site 850, and theclose loop profile will be a triangle (not shown) with the indices of810, 815 and 850.

Finally, FIG. 8(f) illustrates an example in which the centrally locatednewly harvested follicular unit site 885 has five satellite sites 860,865, 870, 875 and 880 around it. Rather than providing a visualrepresentation of an exclusion zone that only comprises a simple circlessurrounding the newly harvested follicular unit site 885, the processor,having determined that each of the already existing harvested follicularunit sites 860, 865, 870, 875 and 880 surrounding the centrally locatednewly harvested follicular unit site 885 are less than or equal to, inthe provided example, twice the minimum harvesting distance from it,forms a polygon 890 (supplemental exclusion zone) linking all thesatellites to surround the centrally located newly harvested follicularunit site 885. In this instance, by combining this polygon shape 890with the six circular exclusion zones corresponding to the alreadyexisting harvested follicular unit sites 860, 865, 870, 875 and 880, nogaps are visualized within the visual representation of the modifiedexclusion zone. This potentially enables a reduction in computerprocessing time, and it also provides benefit to the user by ultimatelyenabling the user to more easily identify on a display and focus on thelarger gaps that may exist.

FIG. 9(a) illustrates a visual representation which only utilizes amethodology in which the exclusion zones are centered about theharvested follicular units. In FIG. 9(a) attention is drawn to oneparticular gap 910. On the other hand, FIG. 9(b) illustrates the visualrepresentation which utilizes a combined methodology in which theexclusion zones are centered about the harvested follicular units andthen combined with the overlay of the supplemental exclusion zoneprovides by the appropriate satellite sites. As seen in FIG. 9(b), thelocation where there used to be a gap 910 is no longer there. A visuallymore “friendly” image has been rendered. In summary, according to someembodiments a method is provided for generating a visual representationof a region where a procedure was performed. The method comprisinggenerating a visual representation of one or more procedure sites (e.g.harvest sites) where a procedure was performed. If more than oneprocedure site exists, the visual representation of the exclusion zonesfor each of the procedure sites are overlapped. The method furthercomprises generating one or more supplemental exclusion zones for any orall procedure sites; and overlapping the exclusion zone and thesupplemental exclusion zones to generate the representation of theperformed procedure region. The step of generating one or moresupplemental exclusion zones may be accomplished by comparing a distancebetween a particular procedure site and one or more previous proceduresites that surround that particular procedure site, and for thosesurrounding sites where the distance is within a predetermined orselected limit, such surrounding sites are identified as the satellitesites for the particular site. In some embodiments, the above-mentionedcomparison may be run against every existing procedure site to identifya corresponding collection of the satellite sites for each existingprocedure site. In some embodiments, with reference to hairtransplantation, each new harvested hair graft may be compared to any orall other previous harvested hair grafts and be added, as appropriate,based on the results of the comparison, to a collection of the satellitesites for each relevant previous harvest site. Alternatively or inaddition, any or all previous harvest sites may be compared to a newlyharvested site and, based on the results of such comparison, beidentified as the satellite sites for the newly harvested site. In someembodiments, the satellite sites may be sorted based on certaincriteria, such as the tangle angle in the coordinate system, forexample, with the smaller angle going first, forming a counter-clockwisesequence, or with a greater angle going first, forming a clockwisesequence. This sorting may be used to generate a continuous convexprofile. Without sorting, the random sequencing of the satellite sitesmay miss some parts of the geometry profile. In some embodiments, themethod may further comprise updating and displaying the previousprocedure region.

It may be desirable in various procedures to identify “reserved regions”where procedure should not be performed. These reserved regions will bedescribed in reference to hair harvesting and implantation and thereforewill be referred to the “reserved harvest regions”, however, it shouldbe understood that this description applies to various “reservedregions” for various medical procedures within the scope of theinventions described herein. Reserved harvest regions define areas fromwhich hairs are not to be selected for harvesting. These reservedharvest regions may define areas where skin conditions exist that makethe area unsuitable or undesirable for harvesting from or implantinginto, areas which contain previously implanted follicular units, areascontaining a particular classification of follicular unit (such as F1for example) that are not desired for the current harvest, areas wheremoles or scars exist, or define areas exhibiting any number of otherconditions. These reserved harvest regions can be illustrated, as shown,for example, in FIG. 10, as a box 1010, or as a circular representation(note that if a circular representation is used, the circlesrepresenting the reserved harvest regions may be formed in a differentcolor than the circles used to identify the exclusion regions 1020), oras any arbitrary shape, and may be created in numerous ways. Forexample, the user may manually define a reserved harvesting site bymanually clicking a mouse at a point within the revised boundary 410, tocreate a reserved harvest region box of pre-defined size. Alternatively,a reserved harvest region box may be created whose sides may be adjustedby the user, or several points may be identified by the user, and theprocessor may create a closed-loop arbitrary shape that encompasses allthe identified points. In an alternative, the reserved harvest regionsmay be automatically created by the processor once it has processed theinformation contained in the acquired images, and the user may beallowed to accept or reject these automatically created reserved harvestregions. It will be apparent that there are many other ways in whichsuch reserved harvest regions may be created for or by the user.

Returning to the discussion of the fiducials, sometimes not all thefiducials are visible in the frame of view of the camera. For example,there may be situations in which all of the fiducials are not visible,and only a subset of them is. In this embodiment of the invention, thesystem may use the limited information initially available andultimately create a register of the location of all the fiducials withrespect to each other.

According to another aspect of the present application, examples oflocating and registering a plurality of fiducials are described inreference to FIGS. 5(a)-5(g). With reference to the example of therobotic system of FIG. 2, since the field of view of the cameras (forexample, 2 cameras used for stereovision) may be smaller than the areaspanned by the fiducials, it may be necessary to move the robot aroundthe boundary (e.g. perimeter of the tensioner or a region defined by apattern of existing hairs acting as fiducials) to capture the locationsof the fiducials. This motion may be performed, for example, manually byan operator's hand dragging the imaging mechanism attached to therobotic arm using “force-control”, or by manipulating the roboticpendant. However, in a preferred embodiment, a robotic arm with theattached image acquisition device may be moved automatically around theperiphery of the skin tensioner (or around the boundary with a pluralityof fiducials that defines the harvesting or implantation region). In theautomated approach, the robot may be first moved manually to the initialposition which brings enough fiducials into view to establish thefiducial frame of reference. Typically, it requires at least three (3)fiducials. FIG. 5(a) illustrates an initial frame of view 502 taken fromthe frame of view of the camera(s) which is mounted on a robotic arm ofa follicular unit harvesting or implanting system, the frame of viewhaving a center point of reference 504. In this embodiment, fourfiducials A, B, 1 and 2 are visible in the initial frame of view 502. Inorder to utilize the teachings of this invention, the system has toacquire the location and/or orientation of each fiducial with referenceto at least one other fiducial. As mentioned above, in order to obtaininformation pertaining to both location and orientation, at least threefiducials are required to be visible in the initial frame of view 502.For example, some examples of obtaining, tracking and recordinginformation about fiducials that could be used in the present inventionis described in the commonly owned co-pending patent applicationpublished as US 2010-0080415A1 on Apr. 1, 2010, which is incorporatedherein by reference.

To enable the system to acquire the location and optionally theorientation of the other fiducials, the system initially moves the fieldof view of the camera over the body surface such that one of thefiducials that was in the initial frame of view 502, is located at thecenter of the frame of view, that is, that the centroid of fiducial 1 issubstantially aligned with the point of reference 504, as shown in FIG.5(b). This first fiducial 1 is allocated location and optionallyorientation coordinates, for example it may be given the referencecoordinates of (2,4). The image processor subsequently identifies thenext closest fiducial that has not already been centered. In the eventthat there are two or more closest fiducials, the system is configuredto select the closest fiducial according to a predetermined selectionmechanism. The selection mechanism may be relatively simple, such asalways selecting the one towards a particular direction, and only doingso if the reference coordinates of the fiducial in that direction havenot already been acquired. In this instance, the selection mechanismhierarchy may comprise, for example, the order of to the right,downwards in direction, toward the left and finally upwards indirection.

FIG. 5(c) illustrates the camera having been moved over the body surfacesuch that the centroid of the closest fiducial 2, to the right hand sideof the fiducial 1 is located at the reference location 504 of the fieldof view 502. In order to get to this particular location, the movementundertaken by the camera itself is registered. For example, if thecamera movement from a position where the centroid of fiducial 1 was atthe point of reference 504 to a position where the centroid of fiducial2 was at the point of reference 504, is defined by (2,0), then thecoordinate for the location of the centroid of fiducial 2 would be (4,4). Similarly, in FIG. 5(d) the camera movement from a position wherethe centroid of fiducial 2 was at the point of reference 504 to aposition where the centroid of fiducial 3 was at the point of reference504, as (2,0), then the coordinate for the location of the centroid offiducial 3 would be (6, 4). It can be seen that in FIG. 5(e), once thecentroid of fiducial 4 is placed at the point of reference 504, andallocated the coordinates (8,4) there is no fiducial to the right, andin FIG. 5(f), the camera is moved (2,−1) such that the centroid offiducial D is at the point of reference 504. In this manner, thelocation of the centroids of all fourteen distinguishable fiducials(shown in FIG. 5(g)) are known with respect to each other. In possessionof this information, in some embodiments virtual lines 506, 508, 510 and512 may be drawn to define an initial boundary, and after taking intoaccount the location of the centroids of the fiducials, for example,from the inner edges of the tensioning device, the relative height ofthe tensioning device above the body surface (assuming an embodimentwhere there is one) and/or the tool diameter, a revised boundary 514 canbe determined, inside of which the coordinates of follicular unit can beidentified for harvesting, or the coordinates of follicular unitimplantation sites can be identified for implantation.

According to one embodiment of the method of the invention, an initialimage and one or more successive images are taken of a skin surfacecontaining a plurality of fiducial marks. In each successive image, theoffset of the fiducials from their positions in the initial image isrecorded by computing, for example, a best-fit transformation T thatminimizes the error between the original positions and the transformedvalue of the subsequent positions. If new fiducials are seen insubsequent images, their positions are transformed by the inverse of Tso that they too can be recorded in the frame of reference of theinitial image. Once their location is transformed and recorded, thesenew fiducials can then be used in conjunction with the originalfiducials to locate an update to the best-fit transformation T. Thisfiducial offset information is utilized in processing the locationand/or orientation, for example, of a harvesting site, applying theoffset to the intended harvesting location prior to carrying out theharvesting itself. Similarly, the fiducial offset information could beused in processing locations and/or orientations of the intendedimplantation sites and such offset could be applied to the intendedimplantation location prior to actual implanting.

Having created a set of coordinates for carrying out the harvesting orimplanting procedure, as long as a couple of fiducials can be seen inthe frame of view, the procedure can be carried out, using the visiblefiducials as reference points. In the case where the field of view isisolated from the fiducials, harvesting locations from where follicularunits have already been harvested or implantation sites into whichfollicular units have already been implanted can be used to supplyadditional reference points, to which future harvesting or implantationlocations can be referenced.

In one example of the embodiment of the invention, a method is providedthat allows defining a region over which a tool is to be operated, forexample, to harvest or implant hair grafts. In one preferred embodiment,such method may be substantially automated (which means that at leastmost of the steps could be performed automatically by the robotic systemitself). It does not exclude that a user may intervene and participate,for example, by giving an alternative command through a user interface,or override the automated command. Generally, if a robotic system,similar to a system shown by example in FIG. 2 is used, an operator mayinitiate an automatic procedure as follows. One of the fiducials thatwas previously identified may be arbitrarily chosen. The robotic armwith a camera operatively connected to it may move automatically tocenter the fiducial in the field of view of the camera. As long asspacing between the fiducials is less than half of the field of view,this will assure that at least two fiducials will be visible. Theprocessor may then direct the robotic arm with the camera to choose thenext closest fiducial that has not already been centered. In the samefashion, the robotic arm will continue to move automatically to the nextfiducial to center the next fiducial until all fiducials have beenidentified. If the fiducials are located on the skin tensioner, then therobotic arm will be directed to move around the skin tensioner. Once allrelevant fiducials are registered in the fiducial frame of reference, ifdesired, the boundary of the region defined by the fiducials may beautomatically computed by the processor, and furthermore such boundarymay be adjusted so that a tool (e.g., harvesting needle) may safelyaccess follicular units inside the boundary. The start and the initialharvest position and direction may be computed automatically by theautomated system. To minimize any potential interference of salineand/or blood in the field of view, the processor may be programmed tostart harvesting or implanting from the edge of the boundary with thelowest height and close to the corner. The robotic arm is directed toautomatically move with the attached tool to harvest or implant hairgrafts within the boundary.

FIGS. 6(a)-6(f) demonstrate an embodiment according to anotherimplementation of the invention which uses a defined virtual selectionregion. The tool may be moved automatically within such selection regionas explained below based on desired criteria. Let us assume, for thepurposes of this particular discussion, that the revised boundary 610has dimensions, for example, in the region of 4 cm horizontally and 3 cmvertically. Having established coordinates of the fiducials (such asfiducials 605 illustrated in FIG. 6a ), as described in earlierexamples, the tool is operated (for example, automatically orsemi-automatically) to initiate the harvesting procedure from the bottomleft hand corner of the revised bound area. For example, the tool may beoperated to move to the location that is approximately at theintersection of row C-F and column 1-5, and aligned with the follicularunit 615. The processor at this time may also dictate that the tool bemoved in the general direction of arrow 620, away from the location offiducial C and towards fiducial F, in a horizontal directionsubstantially parallel to a horizontal side of the revised boundary 610.Based on the exact coordinates of the tool's location with respect tothe image frame of reference, the processor may compute virtualboundaries of a smaller virtual selection region 625 located just infront of the tool in the direction of travel 620. In this particularillustrated example, the virtual selection region 625 may comprise aquadrilateral, such as a rectangle having, for example, dimensions of6-8 mm by 3.5-4.5 mm. Other dimensions of the selection region 625 arealso contemplated within the scope of this application. Use of a smallervirtual selection region 625 reduces the computation required to find asubsequent follicular unit to harvest by restricting the area ofconsideration to an area just in front of the previous harvestedfollicular unit 615 and along the direction of travel 620. The tool isoperated to harvest the follicular unit 615 and the location of theharvested follicular unit 615 is visually identified, for example, by acircle 630, as seen in FIG. 6(a). After harvesting the follicular unit615 the harvesting tool is moved in the general direction of the arrow620, and operated to harvest one or more follicular units located withinthe virtual selection region 625. As seen, there are several follicularunits located within the region 625. However, the next selectedfollicular unit for harvesting may be not the follicular unit locatedwithin the shortest distance from the harvested follicular unit 615inside the region 625 (such as follicular unit 640), but may instead bebased on predetermined selection criteria, such as in this example wherethe tool is moved to the location of the follicular unit 635 that is theclosest to the horizontal boundary 610.

Examples of a few criteria that could be used in directing movement ofthe tool within the selection region (such as region 625 of FIG. 6(a))are described with reference to FIGS. 7(a)-7(e) below, but it will beappreciated that many other criteria may be chosen or predetermined, andsuch criteria may vary during the hair transplantation process. In FIGS.7(a) and 7(b), three follicular unit s have been identified within theselection region 625, follicular units 705, 710 and 715. One particularselection criteria may be, for example, such that if follicular unit 705is harvested, the system will be operated to harvest follicular unit715, and leave 710 un-harvested, effectively harvesting every otherfollicular unit. Another or an additional predetermined selectioncriteria, as illustrated in FIGS. 7(c) and 7(d), if follicular unit 720is harvested, may be to harvest every other follicular unit within theselection region except when the distance to the next availablefollicular unit exceeds certain predetermined distance. In the exampleof FIGS. 7(c) and 7(d), even though follicular unit 725 is the nextavailable follicular unit, it is harvested because it is located at adistance, for example greater than 1.9 mm away, from the alreadyharvested follicular unit 720. As seen in FIG. 7(b), once the follicularunit 715 has been harvested, a new virtual selection region 625 iscreated next to the harvested follicular unit in the same direction oftravel. Turning now to FIGS. 7(e) and 7(f), in this illustration, oncefollicular unit 730 has been harvested, follicular unit 735 is leftun-harvested, and although follicular unit 745 is the next availablefollicular unit in the horizontal direction, it too is leftun-harvested. In this example, the predetermined selection criteria isset such that the next follicular unit available can be selected whetherit be the next available closest in the horizontal or the verticaldirection, provided that it is contained within the virtual selectionregion 625. Consequently, follicular unit 750 is harvested, as indicatedin FIG. 7(f).

Returning now to the example we were discussing in FIG. 6(b), it can beseen that the follicular unit 640 was too far in the vertical directionfrom the horizontal boundary 610, and it was therefore not a desirablefollicular unit to harvest at this time. The system, as shown in FIG.6(b) illustrates follicular unit 640 still on the patient's bodysurface, and follicular unit 635 being harvested. On reaching thefiducial F, as illustrated in FIG. 6(c), the image processor ascertainsthat the revised vertical boundary has been reached, and provides thecontrol signals necessary for the robotic arm to move in the directionof arrow 650, as shown in FIG. 6(d). If desired, the movement of thetool in the direction of arrow 650 will allow the harvesting tool toharvest the follicular unit 640 that was previously left un-harvested.In FIG. 6(e), it can be seen that the virtual selection region 625 ismoved in the direction 650 along the row F-C until all desiredfollicular units in that row are harvested. When no desired follicularunits remain to be harvested within the desired “harvest quadrilateral”along the current row, the tool is operated to move in an upwardlydirection 660 and towards the right, in the direction of arrow 665, toharvest follicular units in the row B-E in a similar manner, asillustrated in FIG. 6(f). Since hair and fiducials are in the sameframe, it can be computed to determine whether the current harvest rowneeds to be incremented: move to the new row when there are no remaininghairs in the current row within the “harvest quadrilateral” formed bylines a specified distance away from the rows of fiducials. In theexample of the robotic application, as the robot moves an automatedharvesting (or implanting) tool along a current row and approaches acorner along the edge or boundary, the robotic system reverses directionand starts searching along a row spaced a configurable distance towardthe opposite edge.

Although the embodiment illustrated and described above with respect toFIGS. 6(a)-6(f) describes that follicular units are harvested within thevirtual selection region until no desired follicular units remain to beharvested within the desired “harvest quadrilateral” whereupon thevirtual selection region is incremented to the next row, it isunderstood that this is an example. Other procedures are possible andcontemplated without departing from the scope of the present disclosure.

By way of an example, in some implementations, follicular units may becollected by proceeding along a row and then automatically incrementingto the next row. However, in some cases, a harvest target of harvestinga particular percentage of follicular units within an area of skin maybe established. For example, a harvest target of harvesting 50% of thefollicular units within an area of skin may be established. In order forthis harvest target to be achieved, a follicular unit row target ofapproximately 10 follicular units may need to be harvested within eachrow into which the area is divided. However, if the row is automaticallyincremented when the end of a row is reached, the follicular unit rowtarget of 10 follicular units (and thereby the harvest target of 50%)may not be achieved.

In another example, the area may be divided into rows and follicularunits within a row (such as the row located at the bottom of the area)may be harvested (such as within a virtual selection region moved alongthe row and/or moved back and forth along the row) until a particulartarget of the numbers of the follicular units for the row is reached. Insome cases, the determination as to whether or not the follicular unitrow target (such as 10 follicular units) for the row is reached may bemade at the end of the row. In other cases, the determination may bemade at other times, such as subsequent to each time a follicular unitis harvested. Regardless, if the follicular unit row target has not beenreached, harvesting continues within the row. However, if the follicularunit row target has been reached, harvesting may continue at the nextrow.

Although this example describes incrementing the row from whichfollicular units are to be harvested only if the follicular unit rowtarget for the row has been precisely met, it is understood that this isfor the purposes of example. In other implementations other proceduresare possible and contemplated without departing from the scope of thepresent disclosure. For instance, in some cases, a certain number thatis less than the follicular unit row target may be harvested from one ormore rows of the area while still achieving the overall desired targetnumber for the area. In such cases, the row may be incremented if thenumber of follicular units that have been harvested is within a range ofthe follicular unit row target for the row, or a desired percentage ofan area of skin.

For example, a harvest target of 75% may be set for an area of skin. Toachieve the harvest target, an average of 15 follicular units may needto be harvested from each row into which the area has been divided, somerows providing more than average 15 and some less than average 15follicular units, as long as the actual number of follicular unitsharvested in the relevant area averages 15 follicular units per row. Inanother example, during processing of a row, a comparison may be madebetween the number of follicular units that have been harvested and thefollicular unit row target of 15. A threshold range above and below thetarget number may be established in certain embodiments. If the numberis above a lower threshold value (such as within three follicular unitsof the target 15, or at least 12), the row may be incremented. However,if the number is below the lower threshold (less than 12 if thethreshold is three follicular units), harvesting may continue within thecurrent row. Similarly, the row may be incremented when the upperthreshold value of the range of the desired target number is achieved.

The above description of incrementing rows is discussed within thecontext of harvesting follicular units. However, it is understood thatthis is for the purposes of example and such row incrementing is notlimited to harvesting of follicular units. In various implementations,such techniques may be used in the context of transplanting follicularunits, other medical procedures, and so on without departing from thescope of the present disclosure. Further, although the above descriptionrefers to ‘rows,’ it is understood that a row as discussed herein doesnot refer to a straight line. A ‘row’ may be any portion of a selectionregion of some width and follicular units may be positioned within suchrow in a way that is not uniform (i.e., follicular units may bepositioned slightly higher than others, slightly lower than others, atvarious distances from each other, and so on).

The embodiments illustrated and described above with respect to FIGS.6(a)-6(f) and FIGS. 7(a)-7(e) illustrate a number of criteria forselecting follicular unit harvesting or implantation sites. However, itis understood that these criteria are provided for the sake of exampleand are not intended to be limiting. Follicular unit harvesting orimplantation sites may be selected according to a number of differentmethods without departing from the scope of the present disclosure.

FIGS. 12(a)-12(h) illustrate a number of different methods for selectingfollicular unit harvesting or implantation sites in order to closelypack such follicular unit harvesting or implanting sites in accordancewith various embodiments of the present disclosure. FIGS. 12(a)-12(h)are illustrated and described below as involving harvesting offollicular units. However, it is understood that this is for thepurposes of example and the techniques disclosed may be utilized in thecontext of implanting follicular units and/or other medical procedures.

FIG. 12(a) illustrates follicular units 1250 on the skin of a patientwithin a boundary 1210. Circles 1260, 1201, 1202, and 1203 illustratesites where follicular units have already been harvested. Within theboundary 1210, follicular units for harvesting may be made from within avirtual selection region 1225 which may be moved, for example, from leftto right across rows beginning at the bottom left and proceeding to thetop right of the boundary 1210. As illustrated, a number of follicularunits (including 1231, 1232, and 1233) are within the virtual selectionregion 1225 and are therefore candidates for harvesting.

In some cases, follicular units may be selected using a ‘lowest andclosest’ method. The lowest and closest method may select follicularunits that are the lowest in the virtual selection region 1225 andclosest to the current position of the harvesting tool in order tominimize harvesting tool movement in order to harvest follicular units.The harvesting tool may be aligned with the bottom left of the virtualselection region 1225. In FIG. 12(a), the follicular unit 1232 may bethe lowest and closest follicular unit to the bottom left of the virtualselection region 1225 (i.e., the current position of the harvestingtool). As such, the follicular unit 1232 may be selected and harvested(as is illustrated by the circle 1240 in FIG. 12(b)).

However, selection of follicular units using the lowest and closestmethod may not result in a particularly close packing of harvest sites(i.e., some of the closely located follicular units may be ignoredbecause they are not “the lowest” which will result in less than desirednumber of the selected follicular units). To improve the packing of theharvest sites, for example, to achieve the higher number of theharvested or implanted follicular units within the row, in variouscases, follicular units may be selected using various enhancements,including without limitation an ‘overlap priority’ method, a ‘positionpriority’ method, a pattern-based method, such as ‘triangular patternpriority’ method, and/or a combination of these methods. It isunderstood that any of these methods and/or combination of these methodsmay also use the lowest and closest method to select between multiplecandidates identified by the respective method or combination of themethods. Such methods may result in a closer packing of harvesting sitesthan selection utilizing the lowest and closest method.

According to an example of the ‘overlap-based’ or ‘overlap priority’method, exclusion zones may be identified around previous harvest sitesinside which follicular units will not be selected. Potential exclusionzones for follicular unit harvesting candidates may also be identified.Overlap between the existing exclusion zones for already harvestedfollicular units and the potential exclusion zones for the futurecandidates follicular units may then be analyzed to select or eliminatecertain follicular unit harvesting candidates.

FIG. 12(c) illustrates exclusion zones 1271, 1272, and 1273 aroundprevious harvest sites 1201, 1202, and 1203, respectively. FIG. 12(c)also illustrates potential exclusion zone 1274 a around harvestingcandidate 1233, potential exclusion zone 1274 b around harvestingcandidate 1232, and potential exclusion zone 1274 c around harvestingcandidate 1231. As FIG. 12(c) shows, only potential exclusion zone 1274a overlaps in any way with any of the exclusion zones 1271, 1272, or1273, while the harvesting candidate 1233 itself is not within any ofthe exclusion zones 1271, 1272 or 1273. As such, the follicular unitharvesting candidate 1233 is selected for harvesting (as is shown inFIG. 12(d)). In some applications, for example, the candidate follicularunit that does not fall within the exclusion zone of a harvest site, andhas the most overlap between the potential exclusion zone and theexisting exclusion zone may be selected for harvesting.

The following example formula may be used in some implementations todetermine whether or not the exclusion zone of a harvesting candidatehas overlap with an exclusion zone from a previous harvest site:

${Objective} = {\sum\limits_{{{if}\mspace{14mu}{{distance}{(i)}}} < {2*{minDistance}}}\;\left( {{2*{minDistance}} - {{distance}(i)}} \right)^{2}}$

In the above example formula, minDistance may be a pre-set value (suchas 1.6˜2.0 mm) to define, for example as shown in FIG. 12(c), the radiusof a circle, for example, from a center of the previous harvest site. Ifthe distance(i) between a candidate harvesting site and the previousharvest site is less than 2 times minDistance, the two circles (eachcentered on the respective follicular unit, or location of previouslyharvested follicular unit, with radius of minDistance) have overlap. Theformula determines how much the overlap is. If distance is small, thisdetermined overlap will be higher. If distance is large, for example, 2times minDistance, the value will be 0, which means no overlap. Asstated above in the formula, as long as distance (i) is less than 2times minDistance, the formula is useful to determine how much overlapexists and if there are hairs that meet this condition.

However, in some instances, no potential exclusion zones may overlapwith existing exclusion zones. In such an instance, a follicular unitharvesting candidate may be selected utilizing the lowest and closestmethod.

Further, in various instances, a number of potential exclusion zones mayoverlap with one or more existing exclusion zones. In such instances,selection among follicular unit harvesting candidates corresponding tothe overlapping potential exclusion zones may be performed utilizingvarious criteria. In some cases where a number of potential exclusionzones overlap with one or more existing exclusion zones, a follicularunit harvesting candidate may be selected utilizing the lowest andclosest method.

In another case, the follicular unit harvesting candidate thatcorresponds to an overlapping potential exclusion zone may be selectedif that follicular unit harvesting candidate is the lowest in thevirtual selection region 1225 and closest to the current position of theharvesting tool out of all the follicular unit harvesting candidatescorresponding to the overlapping potential exclusion zones. Thisextension/modification of the overlap method may be referred to as the‘overlap-based lowest and closest’ or ‘overlap priority lowest andclosest’ method.

In still another case, the follicular unit harvesting candidate thatcorresponds to an overlapping potential exclusion zone may be selectedif the potential exclusion zone corresponding to that follicular unitharvesting candidate overlaps more existing exclusion zone(s) than anyother potential exclusion zone. This extension to the overlap method maybe referred to as the ‘max overlap priority’ method.

According to another method, such as the ‘position-based’ or ‘positionpriority’ method, a proximity band may be identified around previousharvesting sites. For example, in reference to FIG. 12E, if circularareas corresponding to the exclusion zone for previous harvesting siteshave a 2 millimeter diameter, the proximity band may extend, forexample, 1 millimeter beyond the exclusion zone corresponding to therespective previous harvesting site. Follicular unit harvestingcandidates may then be selected if they are within a proximity band.However, although this example discusses exclusion zones as circularlyshaped as shown in FIG. 12E, it is understood that this is an exampleonly. In various implementations, exclusion zones may be shaped otherthan circles (including the shapes described and shown in reference toother Figures and embodiments) and proximity bands may extend an areaoutside various shaped exclusion zones accordingly.

FIG. 12(e) illustrates proximity bands 1275, 1276, and 1277 aroundexclusion zones 1201 a, 1202 a, and 1203 a of harvest sites 1201, 1202,and 1203, respectively. As FIG. 12(e) shows, only follicular unitharvesting candidate 1231 is located within any of the proximity bands(specifically, proximity band 1275). As such, the follicular unitharvesting candidate 1231 is selected for harvesting (as is shown inFIG. 12(f)).

However, in some instances, no follicular unit harvesting candidates maybe located within a proximity band. In such an instance, a follicularunit harvesting candidate may be selected utilizing the lowest andclosest method.

Further, in various instances, a number of follicular unit harvestingcandidates may be located within proximity bands. In such instances,selection among follicular unit harvesting candidates located within oneor more proximity bands may be performed utilizing various criteria. Insome cases where a number of follicular unit harvesting candidates arelocated within proximity bands, a follicular unit harvesting candidatemay be selected utilizing the lowest and closest method.

In another case, exclusion zones may be identified around previousharvest sites inside which follicular units will not be selected andpotential exclusion zones for follicular unit harvesting candidates thatare located in one or more proximity bands may also be identified.Overlap between the existing exclusion zones and the potential exclusionzones may then be analyzed to select follicular unit harvestingcandidates within one or more proximity bands for selection (such as bythe overlap priority method, the overlap priority lowest and closestmethod, the max overlap priority method, and so on). This extension tothe position priority method may be referred to as the ‘position-basedoverlap’ or ‘position priority overlap’ method.

Additionally, though the position priority overlap method firstdetermines follicular unit harvesting candidates that are located withinproximity bands and then determines which of these have potentialexclusion zones that overlap with existing exclusion zones, it isunderstood that this is for the purposes of example and is not intendingto be limiting. In a ‘overlap priority position’ method, follicular unitharvesting candidates that have potential exclusion zones that overlapwith existing exclusion zones may first be determines and then which ofthese are located within proximity bands may be determined.

The following example formula may be used to combine overlap-based andposition-based methods:

${Objective} = {{\sum\limits_{{{if}\mspace{14mu}{{distance}{(i)}}} < {2*{minDistance}}}\;\left( {{2*{minDistance}} - {{distance}(i)}} \right)^{2}} - {w*\left( {{{ratio}*Y} + X} \right)}}$

In the above example formula, X and Y may be the relative distance in Xand Y to a candidate follicular harvesting site. Ratio may be a factordefined to assign more weight in the formula to the X axis or the Yaxis. W may be a weight to select between overlap priority or positionpriority methods. Regarding the term [−w*(ratio*Y+X)] of the formula, ifa candidate follicular harvesting site is closer to a previous harvestsite (smaller Y and X), this value will be large. However, if acandidate hair is further away (larger Y and X), this value will besmall (possibly negative). Again, as stated above in the formula, theformula is useful when distance (i) is less than 2 times minDistance.

Yet other methods contemplated by the present disclosure may be‘pattern-based’ or ‘pattern priority’ methods. For example, one suchpattern-based method may be a ‘triangular pattern-based’ method or‘triangular pattern priority’ method. In the triangular pattern prioritymethod, for example, an equilateral triangle may be formed with a baseof a triangle being a distance between two previous harvesting sites(e.g., distance “x”). An equilateral triangle is a triangle thatincludes sides of all the same length. Once a third point or apex of theequilateral triangle (other than two previous harvest sites) isdetermined, any hair that is positioned within a predetermined smalldistance (such as, for example, one half of “x”) may be selected forharvesting. Alternatively, in other embodiments several triangles may beformed between two previous harvesting sites and available candidatefollicular units. One triangle may be closer to an equilateral trianglethan another triangle, even if neither has sides of all the same length,if the differences between the sides of the first triangle is smallerthan the differences between the sides of the second triangle. Forexample, a first triangle with sides 5-6-7 is closer to an equilateraltriangle than a second triangle with sides 5-14-22. Among availablecandidates, one would give priority to the candidate follicular unitswhich forms triangle that is closest to the equilateral triangle thantriangles formed by other candidates and previous harvesting sites.

FIG. 12(g) illustrates equilateral triangles 1282 and 1283 respectivelyformed based on previous harvest sites. In the illustrated example,triangle 1283 is formed based on the distance between two previousharvesting sites, as shown. Any candidate follicular unit that ispositioned, for example, within a certain small distance (which could bepre-determined, or selected by the user, and in some embodiments, may bea percentage of the distance between the two previous harvesting sitesthat formed the basis for the triangle) from an apex of such trianglemay be selected for harvesting. As seen in FIG. 12(g), candidatefollicular unit 1232 is not within a predetermined distance from theapex of the triangle (shown as shaded area), and therefore is notselected for harvesting. On the other hand, as FIG. 12(g) shows,follicular unit candidate 1233 is positioned close to or within a smalldistance from the apex of the equilateral triangle 1282. As such,follicular unit harvesting candidate 1233 is selected for harvesting (asis shown in FIG. 12(h)).

The following example formula may be used to determine triangles betweenfollicular harvesting candidates and previous harvest sites:Objective=(distanceA−minDistance)²+(distanceB−minDistance)²

In the above example formula, distanceA (which may correspond to a linedefined as a point C to a point A of a triangle) and distanceB (whichmay correspond to a line defined as point C to a point B) may be the twoshortest distances from a candidate follicular harvesting site(corresponding to the point C) to previous harvest sites (correspondingto points B and A). If points A, B and C form a triangle with equal edgedistances, minDistance may be close to both distanceA and distanceB.

However, in some instances, no triangle may be identified betweenfollicular unit harvesting candidates and previous harvesting sites. Insuch an instance, a follicular unit harvesting candidate may be selectedutilizing the lowest and closest method.

Further, in various instances, more than one follicular unit harvestingcandidates may be positioned within a predetermined distance from theequilateral triangles, or correspond to triangles that are approximatelyequilateral triangles. In such instances, selection among follicularunit harvesting candidates that meet the above condition may beperformed utilizing various criteria. In some cases a particularfollicular unit harvesting candidate out of several follicular unitharvesting candidates that meet the above condition may be selectedutilizing the lowest and closest method.

In another case, exclusion zones around previous harvest sites andpotential exclusion zones for follicular unit harvesting candidates thatcorrespond to triangles that are identically close to equilateraltriangles may be identified. Overlap between the existing exclusionzones and the potential exclusion zones may then be analyzed to selectfollicular unit harvesting candidates that correspond to such triangles(such as by the overlap priority method, the overlap priority lowest andclosest method, the max overlap priority method, and so on). Thisextension to the triangular pattern priority method may be referred toas the ‘triangular pattern priority overlap’ method.

In still another case, proximity bands may be identified around previousharvest sites and follicular unit harvesting candidates that correspondto triangles that are identically close to equilateral triangles may beselected if they are within a proximity band. This extension to thetriangular pattern priority method may be referred to as the ‘triangularpattern priority position’ method.

In yet another case, exclusion zones around previous harvest sites andpotential exclusion zones for follicular unit harvesting candidates thatcorrespond to triangles that are identically close to equilateraltriangles may be identified. Overlap between the existing exclusionzones and the potential exclusion zones may then be analyzed to selectfollicular unit harvesting candidates that correspond to such triangles(such as by the overlap priority method, the overlap priority lowest andclosest method, the max overlap priority method, and so on). If multiplepotential exclusion zones overlap existing exclusion zones, proximitybands may be identified around previous harvest sites and follicularunit harvesting candidates that correspond to the overlapping potentialexclusion zones may be selected if they are within a proximity band.This extension to the triangular pattern priority method may be referredto as the ‘triangular pattern priority overlap position’ method.

In still another case, proximity bands may be identified around previousharvest sites and follicular unit harvesting candidates that correspondto triangles that are identically close to equilateral triangles may beselected if they are within a proximity band. If multiple follicularunit harvesting candidates are located within a proximity band,exclusion zones for previous harvest sites and potential exclusion zonesfor follicular unit harvesting candidates within a proximity band may beidentified. If multiple follicular unit harvesting candidates are withina proximity band, exclusion zones around previous harvest sites andpotential exclusion zones for follicular unit harvesting candidateswithin a proximity band may be identified. Overlap between the existingexclusion zones and the potential exclusion zones may then be analyzedto select follicular unit harvesting candidates that correspond totriangles and are within proximity bands (such as by the overlappriority method, the overlap priority lowest and closest method, the maxoverlap priority method, and so on). This extension to the triangularpattern priority method may be referred to as the ‘triangular patternpriority position overlap’ method.

Although the overlap priority method, position priority method,triangular pattern priority method, and various combinations of thesemethods are described above and illustrated in FIGS. 12(a)-12(h), it isunderstood that this is for the purposes of example. Any number of theseand other methods may be combined in any order without departing fromthe scope of the present disclosure. Further, any of the above methodsand/or any combination thereof may use the lowest and closest methodwhen the respective method does not select any follicular unitharvesting candidates and/or when the respective method identifiesmultiple follicular unit harvesting candidates for selection.

Moreover, although FIGS. 12(a)-12(h) are illustrated and described abovein the context of harvesting of follicular units, it is understood thatthis is for the purposes of example. The methods discussed herein may beutilized in the context of implanting follicular units and/or any othermedical procedure.

It will be apparent that the number of steps that are utilized for suchmethods are not limited to those described above. Also, the methods donot require that all the described steps are present. Although themethodology described above as discrete steps, one or more steps may beadded, combined or even deleted, without departing from the intendedfunctionality of the embodiments of the invention. The steps can beperformed in a different order or have the steps shared between morethan one processor, for example. It will also be apparent that themethod described above may be performed in a partially or substantiallyautomated fashion, including performed using robotic systems.

As will be appreciated by those skilled in the art, the methods of thepresent invention may be embodied, at least in part, in software andcarried out in a computer system or other data processing system.Therefore, in some exemplary embodiments hardware may be used incombination with software instructions to implement the presentinvention.

A machine-readable medium may be used to store software and data whichcauses the system to perform methods of the present invention. Theabove-mentioned machine-readable medium may include any suitable mediumcapable of storing and transmitting information in a form accessible byprocessing device, for example, one or more computers. Some examples ofthe machine-readable medium include, but not limited to, magnetic discstorage, flash memory device, optical storage, random access memory,etc.

Certain procedures may require performing the same or similar operationon different areas or portions of the body surface. For example, an areaof the body surface may be divided into several sections and a procedureperformed on one or more sections at time, until the entire area hasbeen covered. For example, during the hair transplantation procedure, askin tensioner may be positioned in a series of positions on thepatient's head, and the hair transplantation procedure performed in eachof the series of positions. In the example of hair transplantationprocedure, this series of positions may be configured to best suit thehair transplantation case in question, but may for example take the formof a grid with two rows and eight columns (four positions on each sideof the head), as illustrated in FIG. 11. The user may place a skintensioner on the patient's head, beginning at the left-hand low corner,move the grid across the patient's head in a series of moves, forexample, 8 moves as shown in the example of FIG. 11, until theright-hand lower corner is reached, and then move up to a second row onthe patient's head, but once again move from left to right, until allsixteen sections have been completed. The advantage of moving from leftto right in this manner is that while follicular units are beingharvested from grid 1115, the dissected follicular units can be removedfrom grid 1110, thereby increasing efficiency of the procedure. For userconvenience to track which grids have had hair harvested from orimplanted into them (or in other applications, grids or sections wherecertain procedure was performed), the monitor 240 can schematicallydisplay the regions to the user.

To enable the system to track which grid location on the patient's headis having the procedure carried out on, the user may be required toprovide some sort of action to enable the system to correlate the gridlocations, in the present example, on the patient's head to the gridlocations on the computer monitor. One way in which the user can providethe identity of the grid location is by selecting the appropriate grid,for example 1110, on the display that corresponds to the location on thepatient's head. Such selection may be provided by clicking of a mousebutton, touching the monitor, or by using the up-, down-, left- andright-arrow keys of a keyboard, for example, or in any number of waysknown to those skilled in the art. By doing this, the system is able toassociate the placing of the skin tensioner in a particular locationwith a designated grid on the display. When the user has selected a gridlocation on the display, the system may also increment a grid numberindicator 1105 on the monitor. For example, when selecting grid 1110,the grid number indicator may indicate that grid 8 has been chosen. Thesystem may then be operated to identify the location of each of thefiducials on the skin tensioner, and to select a location from where thenext hair follicle is to be harvested from, or determine a location intowhich the next hair follicle is to be implanted. When the desired hairhas been harvested from or implanted into the area bound by the skintensioner, for example, using robotic hair transplantation system, theuser may move the skin tensioner to the next grid location, for example1115, on the patient's head, (having first moved the robot to a safelocation so the user can safely access the skin tensioner). Having doneso, the user may once again identify to the system the new grid location1115 on the display. The system will associate the positioning of theskin tensioner with grid 1115 on the display, and increments the gridnumber accordingly, in this case such that indicates grid 9 has beenselected.

The use of grid numbers (in this case 8 and 9) can be used in atreatment report, and allow the physician to correlate dissectionresults to skin tensioner location on the patient's scalp. Knowing whichparameters were used for any one grid location, the user can perhaps tryand optimize the parameters used to provide for optimal harvestingresults. In addition, this also allows the user to select certainparameters that may have been used to one particular grid, and applythem to another. For example, the user may set the system such that onlyevery other hair that is visualized by the imaging system is harvestedfrom grid location 8, and call that particular selection, harvestprogram 1. Rather than having to go through setting all the parametersagain when the skin tensioner is moved to grid 9, the user may simplyselect the same harvesting program that was applied to grid 8, that isharvest program 1, and only every other hair that is visualized by theimaging system will be harvested from grid location 9.

The various embodiments described above are provided by way ofillustration only and should not be construed to limit the claimedinvention. These embodiments are susceptible to various modificationsand alternative forms, and it should be understood that the inventiongenerally, as well as the specific embodiments described herein, coverall modifications, equivalents and alternatives falling within the scopeof the appended claims. By way of non-limiting example, it will beappreciated by those skilled in the art that particular features orcharacteristics described in reference to one figure or embodiment maybe combined as suitable with features or characteristics described inanother figure or embodiment. Further, those skilled in the art willrecognize that the devices, systems, and methods disclosed herein arenot limited to one field, such as hair restoration, but may be appliedto any number of fields. The description, therefore, is not to be takenin a limiting sense, and the scope of the present invention is definedby the appended claims.

It will be further appreciated by those skilled in the art that theinvention is not limited to the use of a particular system, and thatautomated (including robotic), semi-automated, and manual systems andapparatus may be used for positioning and actuating the respectiveremoval tools and other devices and components disclosed herein.Applicant regards the subject matter of the invention to include allnovel and nonobvious combinations and subcombinations of the variouselements, features, functions, and/or properties disclosed herein.

What is claimed is:
 1. A method of operating a tool to harvest orimplant hair grafts, the method comprising: selecting, using at leastone processing unit, a follicular unit harvesting or implanting siteusing at least one of an overlap priority method based on analysis of anoverlap between at least one existing exclusion zone and at least oneproposed exclusion zone, a triangular pattern priority method, a lowestand closest method, or a combination thereof in order to closely packfollicular unit harvesting or implanting sites; and instructing thetool, using the at least one processing unit or at least one differentprocessing unit, to harvest a follicular unit from the selectedharvesting site or to implant a follicular unit into the selectedimplanting site.
 2. The method of claim 1, further comprising: recordingfirst locations of multiple fiducials appearing in one or more images ofa body surface; recording updated locations of at least some of thefiducials in one or more updated images of the body surface; anddetermining an offset of the updated locations relative to the firstlocations of the fiducials for which updated locations were recorded;wherein selecting the follicular unit harvesting or implanting sitecomprises selecting the site based at least in part on the determinedoffset.
 3. The method of claim 1, wherein the overlap priority methodcomprises: identifying at least one exclusion zone around at least oneprevious follicular unit harvesting or implanting site and at least onepotential exclusion zone around at least one candidate follicular unitharvesting or implanting site; and selecting the follicular unitharvesting or implanting site based on overlap between the at least oneexclusion zone and the at least one potential exclusion zone.
 4. Themethod of claim 3, wherein the overlap priority method comprises usingthe lowest and closest method when the at least one exclusion zone doesnot overlap the at least one potential exclusion zone.
 5. The method ofclaim 3, wherein the at least one candidate follicular unit harvestingor implanting site comprises multiple candidate follicular unitharvesting or implanting sites, the at least one potential exclusionzone comprises multiple potential exclusion zones, and selecting thefollicular unit harvesting or implanting site further comprisesselecting one of the candidate follicular unit harvesting or implantingsites that corresponds to one of the potential exclusion zones with thegreatest amount of overlap.
 6. The method of claim 3, wherein the atleast one candidate follicular unit harvesting or implanting sitecomprises multiple candidate follicular unit harvesting or implantingsites, the at least one potential exclusion zone comprises multiplepotential exclusion zones, and selecting the follicular unit harvestingor implanting site further comprises selecting one of the candidatefollicular unit harvesting or implanting sites that corresponds to oneof the potential exclusion zones using the lowest and closest method. 7.A method of operating a tool to harvest or implant hair grafts, themethod comprising: selecting, using at least one processing unit, afollicular unit harvesting or implanting site using a position prioritymethod or a combination of the position priority method with at leastone of an overlap priority method, a pattern priority method, or alowest and closest method, in order to closely pack follicular unitharvesting or implanting sites, and instructing the tool, using the atleast one processing unit or at least one different processing unit, toharvest a follicular unit from the selected harvesting site or toimplant a follicular unit into the selected implanting site, wherein theposition priority method comprises: identifying at least one proximityband around at least one previous follicular unit harvesting orimplanting site, wherein the at least one proximity band is defined byan area starting from a boundary of an exclusion zone corresponding tothe at least one previous follicular unit harvesting or implanting siteand extending a predetermined distance beyond the exclusion zone; andselecting the follicular unit harvesting or implanting site that islocated within the at least one proximity band.
 8. The method of claim7, wherein the position priority method further comprises using thelowest and closest method when no candidate follicular unit harvestingor implanting sites are located within the at least one proximity band.9. A method of operating a tool to harvest or implant hair grafts, themethod comprising: selecting, using at least one processing unit, afollicular unit harvesting or implanting site using a pattern prioritymethod or a combination of the pattern priority method with at least oneof an overlap priority method, a position priority method, or a lowestand closest method, in order to closely pack follicular unit harvestingor implanting sites, and instructing the tool, using the at least oneprocessing unit or at least one different processing unit, to harvest afollicular unit from the selected harvesting site or to implant afollicular unit into the selected implanting site, wherein the patternpriority method comprises: identifying multiple triangles betweenprevious follicular unit harvesting or implanting sites and one of aplurality of candidate follicular unit harvesting or implanting sites;and selecting the follicular unit harvesting or implanting site based onwhich of the triangles is closest in shape to an equilateral triangle.10. The method of claim 9, wherein the triangular pattern prioritymethod further comprises using the lowest and closest method when notriangle can be identified between the previous follicular unitharvesting or implanting sites and the one of the plurality of candidatefollicular unit harvesting or implanting sites.
 11. A method ofoperating a tool to harvest or implant hair grafts, the methodcomprising: selecting, using at least one processing unit, a follicularunit harvesting or implanting site using an overlap priority method,wherein the overlap priority method is based on analysis of an overlapbetween at least one existing exclusion zone and at least one proposedexclusion zone, and selected from the group consisting of an overlappriority lowest and closest method, a maximum overlap priority method, aposition priority overlap method, an overlap priority position method, atriangular pattern priority overlap method, a triangular patternpriority position method, a triangular pattern priority overlap positionmethod, and a triangular pattern priority position overlap method, andinstructing the tool, using the at least one processing unit or at leastone different processing unit, to harvest a follicular unit from theselected harvesting site or to implant a follicular unit into theselected implanting site.
 12. The method of claim 11, wherein theposition priority overlap method comprises: identifying multiplecandidate follicular unit harvesting or implanting sites using theposition priority method; and selecting from the candidate follicularunit harvesting or implanting sites using the overlap priority method.13. The method of claim 11, wherein the overlap priority position methodcomprises: identifying multiple candidate follicular unit harvesting orimplanting sites using the overlap priority method; and selecting fromthe candidate follicular unit harvesting or implanting sites using aposition priority method.
 14. The method of claim 11, wherein thetriangular pattern priority overlap method comprises: identifyingmultiple candidate follicular unit harvesting or implanting sites usingthe triangular pattern priority method; and selecting from the candidatefollicular unit harvesting or implanting sites using the overlappriority method.
 15. The method of claim 11, wherein the triangularpattern priority position method comprises: identifying multiplecandidate follicular unit harvesting or implanting sites using thetriangular pattern priority method; and selecting from the candidatefollicular unit harvesting or implanting sites using a position prioritymethod.
 16. The method of claim 11, wherein the triangular patternpriority overlap position method comprises: identifying multiplecandidate follicular unit harvesting or implanting sites using thetriangular pattern priority method; selecting a group of the candidatefollicular unit harvesting or implanting sites using the overlappriority method; and picking from the group using a position prioritymethod.
 17. The method of claim 11, wherein the triangular patternpriority position overlap method comprises: identifying multiplecandidate follicular unit harvesting or implanting sites using thetriangular pattern priority method; selecting a group of the candidatefollicular unit harvesting or implanting sites using a position prioritymethod; and picking from the group using the overlap priority method.18. A system for hair transplantation, comprising: at least onenon-transitory storage medium storing instructions, and at least oneprocessor that executes the instructions to: select a follicular unitharvesting or implanting site using at least one of an overlap prioritymethod based on analysis of an overlap between at least one existingexclusion zone and at least one proposed exclusion zone, a triangularpattern priority method, a lowest and closest method, or a combinationthereof in order to closely pack follicular unit harvesting orimplanting sites; and instruct a tool to harvest a follicular unit fromthe selected harvesting site or to implant a follicular unit into theselected implanting site.
 19. The system of claim 18, furthercomprising: an interface adapted to receive image data comprising imagesof a body surface; wherein the at least one processor further executesthe instructions to: record first locations of a plurality of fiducialsappearing in one or more images of the body surface; record updatedlocations of at least some of the fiducials in one or more updatedimages of the body surface; and determine the offset of the updatedlocations relative to the first locations of the fiducials for whichupdated locations were recorded; wherein the at least one processingunit selects the follicular unit harvesting or implanting site based atleast in part on the determined offset.
 20. A method of operating a toolto harvest or implant hair grafts, the method comprising: processing atleast one image of a body surface, using at least one processing unit,to divide the at least one image into multiple rows; operating a tool toharvest or implant at least one follicular unit in a first row;determining, using the at least one processing unit or at least onedifferent processing unit, whether a number of harvested or implantedfollicular units in the first row is within a range of a desired numberof harvested or implanted follicular units for the first row; moving thetool based on the results of the determining step as follows: a) if thenumber of harvested or implanted follicular units in the first row iswithin the range of the desired number of harvested or implantedfollicular units for the first row, moving the tool to a subsequent row;and b) if the number of harvested or implanted follicular units in thefirst row is less than a lower threshold value of the range of thedesired number of harvested or implanted follicular units for the firstrow, continue harvesting or implanting at least one additionalfollicular unit in the first row.
 21. The method of claim 20, whereinthe determining step is performed either when an end of the first row isreached or whenever a follicular unit is harvested or implanted.
 22. Themethod of claim 20, wherein the lower threshold value of the rangeequals the desired number of harvested or implanted follicular units forthe first row.
 23. The method of claim 20, wherein the desired number ofharvested or implanted follicular units for the first row is calculatedbased at least upon a desired harvesting or implanting percentage for anarea of the body surface corresponding to the at least one image. 24.The method of claim 20, comprising selecting a follicular unitharvesting or implanting site using at least one of an overlap prioritymethod based on analysis of an overlap between at least one existingexclusion zone and at least one proposed exclusion zone, a positionpriority method based on at least one proximity band starting from aboundary of an exclusion zone and extending a predetermined distancebeyond the exclusion zone, a triangular pattern priority method, alowest and closest method, or a combination thereof.
 25. The method ofclaim 24, the method combining the overlap-based and position-basedmethods and assigning a weight to each of the above methods based on adistance from a candidate site to a previous follicular unit harvestingor implanting site.
 26. The method of claim 20, wherein at least one ofthe step of operating the tool or the step of moving the tool iscomputer controlled.
 27. The method of claim 20, the method comprisingdefining a boundary and starting harvesting or implanting from an edgeof the boundary with a lowest height and close to a corner of theboundary.
 28. The method of claim 20, wherein the operating stepcomprises harvesting or implanting within a virtual selection region,wherein the virtual selection region is located just in front of thetool along a direction of travel of the tool from one harvest orimplantation site to a next harvest or implantation site.
 29. The methodof claim 20, the method comprising operating the tool to minimize toolmovement from a current tool position to a position of a next harvest orimplant.
 30. The method of claim 20, further comprising identifying atleast one exclusion zone around at least one previous follicular unitharvesting or implanting site and at least one potential exclusion zonearound at least one candidate follicular unit harvesting or implantingsite; and selecting a follicular unit harvesting or implanting sitebased on overlap between the at least one exclusion zone and the atleast one potential exclusion zone.
 31. The method of claim 20, furthercomprising identifying at least one proximity band around at least oneprevious follicular unit harvesting or implanting site; and selecting afollicular unit harvesting or implanting site that is located within theat least one proximity band.
 32. A system for hair transplantation,comprising: at least one non-transitory storage medium storinginstructions, and at least one processor that executes the instructionsto: process at least one image of a body surface to divide the at leastone image into multiple rows; operate a tool to harvest or implant atleast one follicular unit in a first row; determine whether a number ofharvested or implanted follicular units in the first row is within arange of a desired number of harvested or implanted follicular units forthe first row; moving the tool based on the results of the determiningstep as follows: a) if the number of harvested or implanted follicularunits in the first row is within the range of the desired number ofharvested or implanted follicular units for the first row, moving thetool to a subsequent row; and b) if the number of harvested or implantedfollicular units in the first row is below a lower threshold of therange of the desired number of harvested or implanted follicular unitsfor the first row, operate the tool to harvest or implant at least oneadditional follicular unit in the first row.
 33. The system of claim 32,wherein the at least one processor operates the tool to harvest orimplant the at least one follicular unit in the first row by selecting afollicular unit harvesting or implanting site using at least one of anoverlap priority method based on analysis of an overlap between at leastone existing exclusion zone and at least one proposed exclusion zone, aposition priority method based on at least one proximity band startingfrom a boundary of an exclusion zone and extending a predetermineddistance beyond the exclusion zone, a triangular pattern prioritymethod, a lowest and closest method, or a combination thereof.
 34. Amethod of selecting a follicular unit harvesting or implanting site, themethod comprising: identifying multiple candidate follicular unitharvesting or implanting sites using a triangular pattern prioritymethod; and selecting, using at least one processing unit, a follicularunit harvesting or implanting site from the multiple candidatefollicular unit harvesting or implanting sites using one of an overlappriority method or a position priority method.
 35. A method of selectinga follicular unit harvesting or implanting site using a triangularpattern priority overlap position method, the method comprising:identifying multiple candidate follicular unit harvesting or implantingsites using a triangular pattern priority method; selecting from themultiple candidate follicular unit harvesting or implanting sites, usingat least one processing unit, a group of the candidate follicular unitharvesting or implanting sites using an overlap priority method; andpicking from the group using a position priority method.
 36. A method ofselecting a follicular unit harvesting or implanting site using atriangular pattern priority position overlap method, the methodcomprising: identifying multiple candidate follicular unit harvesting orimplanting sites using a triangular pattern priority method; selectingfrom the multiple candidate follicular unit harvesting or implantingsites, using at least one processing unit, a group of the candidatefollicular unit harvesting or implanting sites using a position prioritymethod; and picking from the group using an overlap priority method. 37.A method of operating a tool to harvest or implant hair grafts, themethod comprising: processing at least one image of a body surface,using at least one processing unit, to divide the image into multiplerows; operating a tool to harvest or implant at least one follicularunit in a first row; determining, using the at least one processing unitor at least one different processing unit, whether a number of harvestedor implanted follicular units in the first row is within a range of adesired number of harvested or implanted follicular units for the firstrow; moving the tool based on the results of the determining step asfollows: a) if the number of harvested or implanted follicular units inthe first row is within the range of the desired number of harvested orimplanted follicular units for the first row, moving the tool to asubsequent row; and b) if the number of harvested or implantedfollicular units in the first row is less than a lower threshold valueof the range of the desired number of harvested or implanted follicularunits for the first row, continue harvesting or implanting at least oneadditional follicular unit in the first row, wherein the at least oneadditional follicular unit is selected by a method comprising combiningan overlap-based and a position-based method, and assigning a weight toeach of the above methods based on a distance from a candidate site to aprevious follicular unit harvesting or implanting site.